Enrique Tomas / Thomas Gorbach / 
Hilda Tellioglu / Martin Kaltenbrunner (Eds.) 


EMBODIED 
GESTURES 


Academic PressWw 


Enrique Tomas / Thomas Gorbach / Hilda Tellioglu / Martin Kaltenbrunner (Eds.) 
EMBODIED GESTURES 


This book has been published in the context of the Embodied Gestures research project financed by the Austrian artistic research 
funds PEEK programme (project number AR-G399). 


Der Wissenschaftsfonds. 


Enrique Tomas / Thomas Gorbach / 
Hilda Tellioğlu / Martin Kaltenbrunner (Eds.) 


EMBODIED 
GESTURES 


TU Academic Pressw 


Cite as: 
Tomás, E., Gorbach, T., Tellioğlu, H., & Kaltenbrunner, M. (Eds.). (2022). Embodied Gestures. TU Wien Academic 
Press. https:/doi.org/10.34727/2022/isbn.978-3-85448-047-1 


TU Wien Academic Press, 2022 


c/o TU Wien Bibliothek 

TU Wien 

Resselgasse 4, 1040 Wien 
academicpress@tuwien.ac.at 
www.tuwien.at/academicpress 


©) © This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License 
(CC BY-SA 4.0). https://creativecommons.org/licenses/by-sa/4.0/ 
BY SA 


ISBN (print): 978-3-85448-048-8 
ISBN (online): 978-3-85448-047-1 
Available online: https:/doi.org/10.34727/2022/isbn.978-3-85448-047-1 


Media proprietor: TU Wien, Karlsplatz 13, 1040 Wien 

Publisher: TU Wien Academic Press 

Editors (responsible for the content): Enrique Tomás, Thomas Gorbach, Hilda Tellioglu and Martin Kaltenbrunner 
Production: Facultas Verlags- und Buchhandels AG 


Proofreading: Ross Ludlam 
Design and layout: Enrique Tomás 


CONTENTS 


Preface 
Contributors 
Acknowledgments 
List of Figures 
List of Tables 


PART | INSTRUMENTAL GESTURE 


1 Creaking apples and singing chairs: On composing with 
objects, actions and sounds 


Cathy van Eck 


1.1 Using an everyday set-up as the corporeal origin for sounds 

1.2 Changing the mapping between the corporeal origins and the 
imaginary origins 

1.3 Composing structures for imaginary origins of sounds 


References 


2 Instrumental infrastructure: Sheet materials, gesture and musical 
performance 


Louise Devenish 


2.1 Introduction 

2.2 Instrumental sheet materials 

2.3 Gesture as compositional material 

2.4 Instrumental infrastructure and notation 
2.5 Conclusion 


References 


Musical References 


xi 
xvii 
XIX 
xxi 


xxiii 


11 


vi 


3  Pre-gesture, gesture and sound on (no) piano: Music from 

Somewhere 
Haize Lizarazu 
3.1 Introduction 
3.2 The invisible: score and pre-gesture 
3.3 The visible: gesture and space 
3.4 Conclusion 

References 


4 Evolving musical expectations: Mutual correlation between a 


CONTENTS 


human musician and an Al musical instrument 


Koray Tahiroglu 


4.1 Introduction 

4.20 . ALterity and the composition Uncertainty Etude #2 
4.3 Achieving musical expectations 

4.4 . Musical expectations as a framework for composition 
4.5 Conclusion 


References 


5 


PART I| EMBODIED OBJECT 


Timescales for sound-motion objects 
Rolf Inge Godgy 


5.1 Introduction 

5.2 Schaeffer’s object focus 

5.3 General object cognition 
5.4 Sound object features 

5.5 Continuity vs. discontinuity 
5.6 Concluding remarks 


References 


6 


Multi-form visualization: A method to compose acousmatic 


music 
Virginie Viel 


6.1 Introduction 


21 


21 
22 
25 
27 


28 


29 


29 
30 
32 
34 
34 


35 


39 


39 
40 
41 
42 
44 
45 


46 


49 


49 


6.2 
6.3 


6.4 


References 


CONTENTS 


6.1.1 — The perceptual process 

6.1.2 Sensory engagement 

6.1.3 The embodied experience 

6.1.4 Musical perception: vectors of qualia 

In my practice: the capacity to imagine 

The use of multi-form visualization for An Ocean on the Moon 
6.3.1 Building the project 

6.3.2 — The structure of the piece 

6.3.3 The spatial composition 

Conclusion 


7 The ocularcentric objectification of musical embodiment in 
cognitive capitalism: Covid-19 as an allegory on the multiple 
senses of touch 


Pavlos Antoniadis 


7.1 
7.2 


7.3 


7.4 


7.5 


References 


Introduction 

The ocularcentric objectification of embodiment in contemporary 
musicology 

Covid-19 as a catalyst for the ocularcentric rendition of embodied 
experience into data in cognitive capitalism 

The last unassimilable frontier in cognitive capitalism: an enactive 
conception of touch 

Conclusion: Zoom walls as a modern allegory on the multiple senses 
of touch 


PART Ill EMBODIED GESTURE 


8 Embodied Gestures: Sculpting sonic expression into musical 
artifacts 


Enrique Tomás, Thomas Gorbach, Hilda Tellioğlu and Martin Kaltenbrunner 


8.1 
8.2 
8.3 
8.4 
8.5 


Introduction: Sonic gestures and acousmatic music 
Sonic gestures, artefacts and embodied cognition 

The Embodied Gestures Project 

Ideating interfaces from spontaneous cognitive mappings 
Interface design 


vii 


49 
50 
51 
52 
53 
54 
54 
56 
59 
60 


61 


63 


63 


63 


66 


69 


74 
75 


81 


81 
83 
84 
84 
90 


viii 


8.7 
8.8 
References 
9 
10 
10.1 
10.2 
10.3 
10.4 
10.5 
References 
11 


CONTENTS 


8.6 


Musical outcomes 

8.6.1 Voices: composition and live performance by Theodoros 
Lotis 

8.62 Magistri Mei - Bruckner: composition for fixed media by 
Jaime Reis 

8.6.3 Improvisation for Embodied Gestures instruments by Steel 
Girls 

Discussion 

Conclusions 


Thomas Gorbach interviews Annette Vande Gorne 


Thomas Gorbach and Annette Vande Gorne 


Gestural and textural approaches with the Embodied Gestures 
instruments 


Theodoros Lotis 


Introduction: gesture and texture 
Voices 

10.2.1 Typology of gestures in Voices 
10.2.2 Typology of textures in Voices 
The C4 

10.3.1 Typology of gestures 

10.3.2 Anatomy of a gesture with C4 
Mapping network 

10.4.1 Overdetermination 

10.4.2 The C4 and weighting distribution in Voices 
Codetta 


Exploring polyphony in spatial patterns in acousmatic music 


Jaime Reis 

11.1 Introduction 

11.2 Space 

11.3 Spatial polyphony 


11.3.1 Asa concept and in practice 
11.3.2 Limits in the perception of spatial polyphony 


91 


91 


92 


93 
94 
96 


97 


99 


107 


107 
110 
111 
112 
112 
114 
116 
117 
117 
118 
120 


120 


123 


123 
124 
125 
125 
126 


11.4 


References 


CONTENTS 


Spatial polyphony in my music 


12  User-centred design as a model-based co-creation process 


Hilda Tellioglu 
12.1 Introduction 
12.2 From sociotechnology as a principle to embodied interaction 
12.3  User-Centred Design as a Model-Based Co-Creation Process 
12.4 Example of a model-based co-creation process 
12.5 Conclusions 

References 


ix 


127 
130 


133 


133 
135 
136 
139 
141 


141 


PREFACE 


We all learned how to communicate way before articulating our first words. Around the 
ninth or tenth month, humans begin to make requests with gestures. Eight months later 
children use both gestures and words at the same time. It is at this age when we find it 
useful to communicate different things with words rather than with gestures. We could 
assert that gestures help us to develop speech. Using gestures, children communicate 
ideas and obtain new words from adults in response to what they are calling for. Adults 
use gestures to communicate their ideas, needs, knowledge and feelings. This is why ges- 
tures are commonly understood as physical movements carrying information. Gestures 
are embodied language and cognition. 

Gestures are well articulated movements. They usually have a clear beginning and 
an end we can easily recognize. Like any other communicative strategy, we can only 
understand what a gesture denotes in the context of a social or cultural practice. For this 
reason, gestures can even produce misunderstandings. 

Gestures are spontaneous acts. It is not easy to identify where a gesture originates. 
Could all bodily movements be interpreted as gestures? Do we concatenate gestures 
while walking on the streets? What transforms the painter’s moving hand into an artistic 
gesture? If gestures are embodied communication, why do we need them when we are 
alone? Gestures reinforce our inner thoughts, fears, reflections, etc. Gestures allude to 
mental imagery and memory. For instance, within the fields of music and visual arts, ges- 
tures are employed to describe musical pieces and artworks. Indeed, we are often invited 
to discover the gestural traces left by the artist with the paintbrush. In a similar fashion, 
Jimi Hendrix’s gestures around his Stratocaster are already part of the social imaginary. 
His gestures carried communication but they mostly served Hendrix in driving his emo- 
tions on stage. These gestures were bonded to a flux of cognitive processes shaping the 
way he played. 

Gestures are so deeply personal that they can be used to elaborate cartoonish repre- 
sentations. We can recognize people from their gestural pose. It seems that human life 
is driven by all these bodily movements we cannot easily control or understand. Just 
like the impossibility for Glenn Gould to forego chanting and whispering while playing 
Bach at the piano. Probably for Gould, musical scores triggered a complex sequence of 
mental images resulting in words, symbols and facial expressions that were absolutely 
necessary to define his sonic outcome. A good example of how much our mind and body 
are embodied in music and vice versa. 
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This book uses the following hypothesis as its departure point: humans tend to de- 
duce gestural and sensorimotor activity from auditory perception. In other words, we 
are inclined to determine both sound context and sound-producing gestures. First, we 
all instinctively have a tendency to identify potential threats from sonic events. Second, 
listening is not a passive action just happening to us. We need to actively learn how to 
give meaning to what we are hearing. In the embodied mind theory perception is not 
something we receive. It is something we actively do. During the action-in-perception 
loop, external stimuli would be incorporated as mental simulations, as reenactments of 
what we perceive. It is particularly important that these simulations can involve senso- 
rimotor activations. That would mean that we all may have an internal representation of 
movement acquired from former experience that can be accessed from various sensory 
information. This cognitive model understands auditory perception partly as simulation 
and control of compatible gestures and bodily movements. We could say that listening 
extends aural information towards sensorimotor action. In fact, the opposite also occurs. 
We often find our feet tapping to rhythmic music. This spontaneous action results from 
our previous encoded sensorimotor simulations. Interestingly, this phenomenon can also 
be observed in other animal species. For instance, parrots who, without training, impul- 
sively tap and shake their heads to the musical tempo. 

The lack of appropriate language for describing auditory events forces us to verbalize 
changes in sound perception though gestures. For instance, people intuitively represent 
sonic changes by tracing imaginary lines in the air. This action—sound tracing—has 
been well studied. High frequencies are usually associated with gestures above our 
head. Low frequency sounds are located somewhere below our hips. Perceived tempo- 
ral changes in sound—often called sonic morphologies—also refer back to sensorimotor 
sensations. 

It is often said that sound and music have the power to create mental images, haptic 
sensations and artistic associations. This possibility affords artistic creation. It is the 
framework where composers and sound artists develop their practice. How has ‘gesture’ 
shaped the way we produce and understand music since the 20th century, especially since 
the experimental electroacoustic period? The development of technologies for audio 
recording, edition and reproduction inaugurated a new era for music research. Recorded 
sound could be edited in pieces and organized in time. In the 1960s, Pierre Schaeffer 
proposed a method for musical composition and analysis that departed from the auditory 
perceptual features of recorded sound. Schaeffer, influenced by Edmund Husserl’s phe- 
nomenology, proposed separating the sound source from its context. Sound become dis- 
regarded from any external significations. Schaeffer built a musical solfége for recorded 
audio based on the notion of ‘sonic object’, the minimal auditive auditory perceptive 
unit, equivalent to the minimal perceivable sonic intention listeners can perceive. Sonic 
objects were characterized and classified depending on their perceptive character. This 
separation between object and subject originated a new kind of perception, the ‘acous- 
matic listening’, decoupled from the context in which the sound was recorded. As a 
reaction, Guy Reibel and Francois Bayle pointed out how this Schaefferian ‘reduction’ 
would interrupt the natural flow of communication between music and listener. For them, 
even in a reduced listening situation, listeners always associate sound with mental im- 
ages, irremediably connected with their physical origin. Yet Reibel and Bayle proposed 
the extension of the notion of ‘sound object’ towards the concept of “sound gesture’. 
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With this strategy, it would be possible to overcome what they called the 'Schaefferian 
Sound Bazar’. They would use the listener’s musical imagination as the channel of com- 
munication between composer and recipient. To this end, Reibel and Bayle proposed a 
compositional methodology based on the extensive use of sound produced as result of 
an action. Indeed, François Bayle, together with Jean-Christophe Thomas, wrote a book 
called Diabolus in Musica, in which around 150 different examples of the composition 
of sonic gestures are analysed and explained. 

Following this methodology, Annette Vande Gorne, founder of the Studio for Musique 
& Recherche in Brussels, has been responsible for the development of new composi- 
tional methods based on the notion of 'sonic archetypes'. She described these major 
archetypes as: percussion-resonance, friction, accumulation of corpuscles, rebound, os- 
cillation, swinging and swaying, flux, pressure-deformation, rotation, and spiral. For 
Vande Gorne, these archetypes would constitute a vocabulary of models, especially con- 
nected with the perception of physical features, which can be used to describe and design 
listening experiences. These sonic archetypes denote a flow of movement and senso- 
rimotor action. Vande Gorne's influential methods can thus be understood as practical 
embodied cognitive research. 

In parallel to Vande Gorne, Denis Smalley proposed a framework to describe the rich 
variety of sonic contents in electroacoustic music. He called it ‘spectromorphology’ and 
it consists of a set of tools for *understanding structural relations and behaviours as ex- 
perienced in the temporal flux of music'. Within this framework, the spectromorphology 
of a musical piece (i.e. temporal spectral flux of music) is mostly discussed in relation 
to gesture. For Smalley, gesture is an energy-motion trajectory creating spectromorpho- 
logical life. Smalley specifically describes how listeners always tend to deduce gestural 
activity from sound and introduces the notion of “gestural surrogacy’, a scale of rela- 
tionships between sound material and a known gestural model (e.g. first-, second- or 
third-order and remote surrogacy). For instance, in his third-order surrogacy level, ges- 
tures are imaged in the music. In the case of ‘remote surrogacy’, music is articulated 
from gestural vestiges. Developing his framework further, Smalley explains that listen- 
ers always attempt to predict the directionality of a morphological change. To illustrate 
this phenomenon, the author describes a sort of image schema (e.g. onsets, continuants, 
terminations) with possible metaphorical interpretations (e.g. for onset: departure, emer- 
gence, anacrusis, attack, etc). Smalley also illustrates processes for typical motion and 
growth processes (unidirectional, reciprocal, cyclic, multidirectional) and texture mo- 
tion (streaming, flocking, turbulence, convolution). Similar categorizations are made in 
relation to spectral and spatial changes. 

Some scholars argue that Smalley's image schemas are implicit embodied cognitive 
theory. The same would apply to Vande Gorne's methods. Under this hypothesis, elec- 
troacoustic and acousmatic music could be considered as embodied cognitive praxis ex- 
tending its current theories. The practice of acousmatic music assumes the mental sim- 
ulation of sound-producing gestures. A key aspect of Smalley's theories is that different 
types of gestures have different embodied-functional associations and, hence, causal dy- 
namics. 

With this book, our intention is visualizing how similar and compatible are the notions 
employed by scientists working in the field of embodied music cognition and the artis- 
tic discourses proposed by musicians working with ‘gesture’ as compositional material. 
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‘Embodied Gestures’ is also the name of the artistic research project developed between 
2017 and 2021 by the editors of this book. Our aim was to study a new paradigm of in- 
terfaces for musical expression especially designed to emphasize a performer's gestural 
embodiment within an instrument. In order to achieve this goal, ‘Embodied Gestures’ 
explored the possibilities of shaping the physical affordances of designed digital instru- 
ments with the intention of inspiring particular forms of gesturality. Specifically, our 
objective was to study the implications of designing musical interfaces that can afford 
the same type of gesturality that a particular sound inspires. For instance, in order to 
control a sound passage composed from the circulation of ‘rotating’ sonic movements 
in space, we designed musical interfaces that afforded by themselves, and through their 
physical affordances, similar ‘rotating’ physical gestures to their performers. 

Throughout the years of this artistic research project, we often found ourselves at 
an intermediate place between embodied music cognition, musical performance practice 
and musical interface design. After this project, we see both Smalley's gestural surrogacy 
and Godgy’s sensorimotor models as complementary explanations towards describing 
the human tendency to deduce sound-producing gestures from what we hear. However, 
artistic practice can often better incorporate experiential aspects of the topics under re- 
search. Artists create knowledge by showing us unexpected realities and embodiments 
of sensory information. In the case of music, composers and musicians can question 
our understanding of why we use gestures, where they originate and their economies of 
production. For this reason, this book compiles manuscripts by researchers who have 
approached the issue of 'gesture' from both the scientific and artistic practice. 

This book is structured in three parts called Instrumental Gesture, Embodied Ob- 
ject and Embodied Gesture. The first section includes four manuscripts. Cathy van 
Eck's ‘Creaking Apples and s Singing Chairs: On Composing with Objects, Actions and 
Sounds’ introduces two pieces by the author: In Paradisum and Empty Chairs. Van Eck 
explores the question of what are actions and objects to make music with, and what not. 
How do movements by the performer relate to what is sounding? And when all kinds of 
actions can be connected to all kinds of sounds due to the use of electricity, how do we 
decide which connections to use? Louise Devenish’s ‘Instrumental Infrastructure: Sheet 
Materials, Gesture and Musical Performance’ discusses the entanglement of instrumen- 
tal materials and gesture as compositional material in contemporary music, a field that 
increasingly explores the sonic and musical potential of a vast range of instruments, ob- 
jects, materials and surfaces. Haize Lizarazu's ‘Pre-gesture, Gesture and Sound on a [no] 
piano: Music from Somewhere’ describes the process of performing a pianist work with 
*no piano'. In it, the author demonstrates how by eliminating the instrument, composer 
and performer are able to focus on those prior moments to the production of visible ges- 
ture, the ‘pre-gesture’. Through this artistic tactic they visualize a clue concept linked 
to memory during the study of a musical piece. The fourth manuscript of the first part 
explores ways to incorporate the notion of embodied gesture into Artificial Intelligence 
and User Interface Design. Koray Tahiroglu’s ‘Evolving Musical Expectations: Mu- 
tual Correlation between a Human Musician and an AI Musical Instrument' provides an 
overview of the current experiential techniques the author has used towards performing 
with artificial intelligence. Both his reflections as an artist and a musician are presented. 
In this article, Tahiroglu refers to a new technological-theoretical framework that can 
offer insight into the creation of compositions for musical instruments that are fully or 
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partly autonomous, or autonomous in an unusual way. Through the composition, Uncer- 
tainty Etude #2, he discusses the use of artificial intelligence methods and a perspective 
from which a creative practice can be proposed to explore unusual musical expectations 
in a music composition. 

In the second part of this book, Rolf Inge Godgy presents *Timescales for Sound- 
Motion Objects”. This article departs from a radical understanding of music as multi- 
modal art, consisting of fragments of combined sound and body motion. Godgy extends 
the classical Schaefferian ‘sound object’ towards the notion of “sound-motion-object’. 
The main idea is that timescales of Schaeffer’s sound objects are closely linked with 
the timescales of body motion, and that thinking multimodal sound-motion objects in 
music would be useful for both analytic and creative work. Virginie Viel’s ‘Multi-form 
Visualization: A Method to Compose Acousmatic Music ’ challenges the instinctive and 
natural tendency to draw connections between sonic and visual stimuli. In this chapter, 
Viel proposes to us an examination of the perceptual process, allowing us to examine the 
perception of qualia in music. Finally, the author demonstrates how conceiving the per- 
ceiver’s mind as embodied could be the key towards the development of compositional 
practices based on the idea of multiform visualization, a personal method developed by 
the author. Pavlos Antoniadis’s ‘The Ocularcentric Objectification of Musical Embodi- 
ment in Cognitive Capitalism: Covid-19 as an Allegory on the Multiple Senses of Touch’ 
is an essay triggered by the Covid-19 pandemic. This paper attempts a problematization 
of the notion of touch in musical performance. The de facto crisis of musical haptics 
due to physical and social distancing is here considered in the context of a wider phe- 
nomenon, namely the ocularcentric objectification of musical embodiment. By revealing 
a crisis of touch, the ongoing sanitary crisis invites us to further reflect on the meaning of 
musical haptics beyond the visual properties of embodied gestures and beyond tactility 
in the design of tangible user interfaces. In that sense, Covid-19 becomes a modern al- 
legory on the multiple senses of touch, similar to the allegories of the senses in Flemish 
Renaissance painting. 

The third part of the book is dedicated to presenting articles produced in the con- 
text of our Embodied Gestures project. In the first of these chapters, Enrique Tomas, 
Thomas Gorbach, Hilda Tellioglu and Martin Kaltenbrunner present ‘Embodied Ges- 
tures: Sculpting Sonic Expression into Musical Artefacts’. In this chapter, the authors 
describe the results of introducing a new paradigm of musical interface design inspired 
by sonic gestures. In particular, the authors discuss the beneficial aspects of incorporat- 
ing energy-motion models as a design pattern in musical interface design. These models 
can be understood as archetypes of motion trajectories that are commonly applied in the 
analysis and composition of acousmatic music. The evaluation through composition and 
performance indicate that this design paradigm can foster musical inventiveness and ex- 
pression in the processes of composition and performance of gestural electronic music. 
The second chapter in this section consists of an interview with Annette Vande Gorne 
conducted by Thomas Gorbach. During a summer workshop in 2020, Gorbach recorded 
the spontaneous and inspirational answers given by Vande Gorne to the question of how 
to engage the notion of gesture in acousmatic music. Additionally, she reflected on the 
possibilities of introducing the idea of energy-motion models into the physicality of the 
instruments employed to produce acousmatic music. Theodoros Lotis’ *Gestural and 
Textural Approaches in Composition and Performance with the Embodied Gestures In- 
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struments” discusses a work composed for “Embodied Gestures instruments’, describing 
how they facilitate the creation of rhythmical structures and looping processes, since 
many of their gestural typologies occur as repetitive patterns. In his opinion, apart from 
the evident approach of the gestural behaviour, the instruments could also be used for the 
control of the spectral/textural evolution (micro- and macro-structural) of sound objects 
or sonic structures. Jaime Reis’ ‘Exploring Polyphony in Spatial Patterns in Acous- 
matic Music’ introduces the readers to the use of space as a parameter in acousmatic 
music—along with some of the main perceptive features involved in such practice. Af- 
ter a brief description of acousmatic music techniques, Reis discusses these relations in 
regard to gestures. Finally, the author describes how he has enhanced spatial polyphony 
through the use of Embodied Gestures instruments in his compositions. Finally, Hilda 
Tellioglu’s *User-centred design as a model-based co-creation process’ reviews the it- 
erative user-centred design (UCD) process as an adaptive and agile life-cycle for open 
innovation and successful development of socio-technical systems. In UCD, the user ac- 
ceptance and usability of a digital system are central and determine the way to proceed 
at several stages of a design process. The question presented in this chapter is how to 
ensure a successfully realized user experience in a new design. In other words, how to 
proceed in a design project to understand the target users and their context, including 
their past experiences, and to consider this insight in the design of artifacts and inter- 
actions provided as part of the new design. The author’s answer to these questions is 
applying modelling in all phases of the design process by creating models of all findings 
gathered after studying the target users and their past and current contexts, as well as by 
preparing and accompanying the design process as a reflective and self-critical practice. 

We hope this book contributes to the ongoing discourse and discussion around creative 
technologies and music, expressive musical interface design, the debate around the use 
of AI technology in music practice, as well as presenting a new way of thinking about 
musical instruments, composing and performing with them. 
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#1 

CREAKING APPLES AND SINGING 
CHAIRS: ON COMPOSING WITH 
OBJECTS, ACTIONS AND SOUNDS 


CATHY VAN ECK 


In my two compositions In Paradisum! (2019) and Empty Chairs? (2018) I use ob- 
jects—apples and chairs—and simple everyday actions—eating the apple and replacing 
the chairs. In both pieces, these actions control a computer patch that produces all kinds 
of sounds, and this combination results in a composition for objects, actions and sounds. 
My two pieces In Paradisum and Empty Chairs came forth from several questions. What 
are actions and objects to make music with, and what not? How do movements by the 
performer relate to what is sounding? And when all kinds of actions can be connected 
to all kinds of sounds, due to the use of electricity, how do I decide which connections 
to use? To elaborate these questions and thoughts on what kind of interdependencies 
I try to develop during my compositions I discuss and compare works by three other 
composers—Nicolas Collins, Chikashi Miyama and Joanna Bailie. 

An important starting point for these pieces is what I would like to call ‘the origins 
of sounds’. These origins can be divided into three categories: the corporeal origins, the 
physical origins and the imaginary origins. I developed these categories because they are 
helpful for my compositional work with movements, sensors and sounds. 

The corporeal origins can be defined as the movements by the musician causing the 
musical instrument to sound. With acoustic instruments this is obvious: a musician 
has to make movements to produce any sounds at all. These might be big movements, 
such as needed when playing the cello, or smaller movements, such as when playing 
the clarinet. Non-electronic instruments each demand a specific physical effort from the 
player (Craenen, 2014, p. 150). Controllers and instruments used to control electronic 


! Some pictures of In Paradisum by Cathy van Eck: https://www.cathyvaneck.net/in-paradisum/ 
(Accessed: 1/12/2021) 

2A documentation of Empty Chairs by Cathy van Eck: https://www.cathyvaneck.net/empty-chairs/ 
(Accessed: 1/12/2021) 


https:/doi.org/10.34727/2022/isbn.978-3-85448-047-1 1. This work is licensed under the 
Creative Commons Attribution-ShareAlike License 4.0 International (CC BY-SA 4.0). 
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sounds also demand a physical effort, albeit often with less constraints. In electronic 
music one can push a button or move a fader to trigger a complex sound. Additionally, the 
relationship between physical effort and resulting sound is not as linear (or even not linear 
at all) as is the case for acoustic instruments. Especially when using digital means, the 
same movement on the same controller could trigger a different sound, depending on how 
a computer or synthesizer is programmed. But also with these kind of set-ups the claim 
Craenen (2014) makes is valid: during sound production ‘the visibility of synchronous 
physical action [...] can augment auditory perception or push it in a certain direction’ 
(p. 150). The movements visually perceptible when someone makes music influences 
how the sound is heard. 

The second category is the physical origins. These are the movements through which 
the sound itself is produced. For acoustic instruments these are, for example, vibration 
by strings, pieces of wood, columns of air, etc. With electronic means these movements 
are nearly always executed by the diaphragms of loudspeakers. For acoustic instruments 
these physical origins are often firmly coupled to corporeal origins. The physical origins 
are defined by the design of the instrument. To bring the instrument into vibration, and 
thus to produce sounds, is achieved by specific movements of the body of the performer. 
The corporeal origins are merely predefined by the instrument itself, and thus by the 
physical origins. It is hard to produce sound by blowing on a violin, and not easy to 
bring out many sounds by hitting a clarinet either. This is different for instruments using 
loudspeakers for producing sound: the corporeal origins are not prescribed by the physi- 
cal origins, but can be designed in many different ways. The physical origins are always 
the same in electronic music: loudspeaker diaphragms vibrating according to an electric 
signal. As Thor Magnusson (2019) states, ‘the electronic or digital instrument has an 
interface, whereas the acoustic instrument is the interface’ (p. 35). For electronic music 
you can choose whether you want to blow, hit, or bow the sound. This is much less easy, 
if not impossible, with acoustic instruments. The same movement always results in the 
same sound. But as I will argue in this text, this unreliability of the relationships between 
corporeal and physical origins in electronic music is an appealing feature of composing 
for movements and sounds. 

The third category is the imaginary origins. Especially in electronic music this cat- 
egory is of much importance. As mentioned above, the physical origins will always be 
the same loudspeaker vibrations, but the imaginary origins of these sounds can be mani- 
fold. When listening to sounds I might recognize different sound sources: I hear drops of 
water falling, someone breathing or footsteps passing by. Evidently I can never be sure 
how these sounds have originally been produced, and that actually does not matter much 
for my perception. As Michel Chion (2010) describes, these imaginary sound sources 
replace the real sound sources, which are the ways in which a sound has been produced 
(p. 45). It is easy to simulate the sound of a train with the help of small objects in front 
of a microphone. Conversely, train sounds could be processed unrecognizably in a piece 
(Chion, 2010, p. 31). The imaginary origins do not have to refer to already known and 
identifiable sounds. These can just as well be abstract synthesized sounds. The recogni- 
tion of imaginary origins is of course partly dependent on the perception of the listener. 
Whereas one person might identify different sounds as being from the same imaginary 
source, another person might identify several different imaginary sources. For composers 
this is of course also one of the big advantages used in many compositions of electronic 
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sound: one can smoothly transform from one imaginary sound source to another. Many 
acousmatic pieces use this as a very important feature of their compositional technique. 
Composers such as Hildegard Westerkamp, Trevor Wishart or Annette Vande Gorne cre- 
ate all kinds of new imaginary origins, which can only exist in sound itself: a voice taking 
flight in Red Bird (Wishart, 1996, p. 166), falling raindrops becoming a musical rhythm 
(Norman, 2004, p. 80), or the buzzing of insects developing into a melodic line (Vande 
Gorne, 2021). 

In the next few examples I would like to investigate how different compositions con- 
nect these different origins, and what kinds of compositional relationships these connec- 
tions create. 


1.1 Using an everyday set-up as the corporeal origin for sounds 


In both my compositions I was interested in investigating objects and actions that have 
the least possible relevance in typical music practices. The objects used should neither 
be played expressively, nor be able to control electronic sounds in a virtuosic way; rather 
they should merely be used as they commonly are in their accustomed environment. I 
envisaged the idea of focusing on very simple actions and using them as the part of the 
set-up that controls the sound production. On instruments, whether standardized or newly 
developed ones, a performer can discover sounds and be able to practice, to control sound 
production very precisely, and eventually become a virtuoso. But I became curious about 
what would happen if I chose the opposite path: looking for an action that is not at all 
suitable for playing an expressive musical performance, which does not give me many 
possibilities for sound control and forces me to move in a certain way. For that reason, 
I chose eating an apple for In Paradisum and moving chairs for Empty Chairs. Both are 
actions with many possible associations. Besides being a very common fruit that is eaten 
worldwide, the apple has also a long cultural history. Eve in paradise, the judgment of 
Paris and the fairy tale of Snow White are a few examples. Chairs are very common 
furniture also used by many on a daily basis. The way chairs are positioned expresses a 
lot about how they are used. In a row, next to each other, in groups of two or in a circle; 
all these positions yield different associations of certain types of social gatherings. 

A beautiful example of controlling electronic sounds with a very common everyday 
action is In Memoriam Michel Waisvisz (2009)? by Nicolas Collins (Figure 1.1). In this 
piece Nicolas Collins uses the light created by the flame of a small candle to control elec- 
tronic sounds. Michel Waisvisz was a composer and instrument inventor who worked a 
lot with all kinds of gesture and sensor interfaces, such as the kraakdoos (Dutch for 
cracklebox) (1974). The kraakdoos is played by making connections with the skin of 
your fingers between different parts of the circuitry. In his In Memoriam Michel Waisvisz, 
Nicolas Collins replaces the lively performer with a small candle, and the circuit is con- 
trolled by photoresistors that change their resistance depending on how much light they 
receive from the candle. Besides the candlelight getting closer the more of the candle has 
burnt down, the position of the flame also influences the sound. To change the position 
of the flame Collins uses a small fan. Four oscillators producing sounds that resemble 


3A video of the work In Memoriam Michel Waisvisz by Nicolas Collins can be watched here: 
https://www.youtube.com/watch?v=sBIIRdnPciw (Accessed: 1/12/2021) 
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Figure 1.1: In Memoriam Michel Waisvisz (version 2015) by Nicolas Collins (video 
still) (Video: ©Henrik Jonsson, 2015) 


the sonic character of a kraakdoos are controlled by the movements of the flame.* As 
Nicolas Collins explains: ‘the reason that circuit sounds the way it does is that the four 
oscillators are tuned in rough unison and when the flame moves around in the field of the 
four photocells it is detuning each against the others’ (Collins 2021). 

What makes the relationship between the movements of the flame and the electronic 
sound very strong is the connection of two phenomena which are commonly associated 
with two very different processes: the burning of a candle—an act reminiscent of the age 
before electricity—and sounds produced by oscillators—not possible without electricity. 
Evidently there is no such thing as a natural cause between the candle and the sound; 
here the electronic circuit design makes the analogue changes in flame and electronic 
sound possible. Even though the flame is of course not a human being, the movements 
of the flame can be regarded here as replacing the movements of a performer. Due to the 
synchronous changes in light and sound the corporeal origins of the sound seem to be the 
movements of the flame. A fascinating and poetic interaction between these two sources 
is the result. 

Whereas the connection between the movements of the candle flame and the sound 
created by the oscillators is continuous, in my two pieces I used a less continuous con- 
nection between objects and sounds. I chose two discrete parts of the process of eating 
an apple for controlling the sound processing: biting into the apple (detected by a contact 
microphone placed upon the apple) and chewing an apple piece (detected by a contact 
microphone on a chewing muscle on my cheek). For the chairs the main parameter con- 


^A description of a similar circuit can be found in Nicolas Collins’ book Handmade Electronic Music (2020, 
p. 100). 
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trolling the sound processing is whether a chair is moving or not. These movements are 
detected and sent to a Max patch on my computer by iPhones attached to the underside 
of the chairs.’ All these controls are either simple triggers (biting and chewing) or on 
and off (chairs moving or not). This kind of control could also be done by a simple 
knob controller. The difference would be in the constraints added to the movements of 
the performer by using these objects. Eating an apple asks for a certain amount of time, 
and whereas a knob can be pushed very often and quickly, each bite of an apple needs 
a certain amount of chews to be swallowed. Chairs are bulky objects, and carrying and 
shoving them around also requires a different timing than pushing a button on a con- 
troller. The timings of these everyday actions become obligatory musical timings in my 
compositional process. 


1.2 Changing the mapping between the corporeal origins and the imagi- 
nary origins 


In In Memoriam Michel Waisvisz the movement of the flame controlled a different elec- 
tronic circuit, and therefore completely different sounds. The relationship between cor- 
poreal origins and imaginary origins can easily be changed, especially when software is 
used instead of hardware. A good example of these kinds of changes in mapping be- 
tween controller and synthesizer can be found in the piece Black Vox (2009) by Chikashi 
Miyama.5 He developed his own instruments because he is ‘not so attracted by tradi- 
tional instruments such as piano or flute, because they are too biased towards the existing 
musical scale’ (Miyama, 2020). He constructed the Peacock, an instrument with 35 in- 
frared sensors that control more than 300 synthesizer parameters in a PureData patch. 
The patch is a phase-bash-algorithm-based synthesizer (Miyama, 2010, p. 381). For 
physical instruments, parameters such as spectrum and pitch or volume and pitch are of- 
ten coupled to a certain degree. Playing louder on a piano changes not only the loudness, 
but also the spectral characteristics of the sound. In Chikashi Miyama’s instrument seven 
sensors in a row control one voice of the synthesizer, resulting in a maximum of five 
voices. Changes in distance to the performer’s body will make their output value lower 
or higher (p. 382). All kinds of parameters of the synthesis process can be shaped by the 
hand and arm movements independently. 

What makes Miyama’s set-up significantly different from conventional instruments is 
that the mapping between body movements and sounds changes during the performance. 
The same hand movement will yield different results at the beginning, in the middle or 
at the end of the piece (Miyama, 2010, p. 382). The imaginary origins thus change, 
although the corporeal origins stay the same. This change of mapping is based on a 
timeline. Miyama can follow a score in his Pure Data patch, which lets him know both 
where in the piece he is and what the current mappings are (Miyama, 2020). Due to 


5There are actually two short moments in the piece where the movement of a chair is controlling an oscillator 
continuously. This is done by tilting the chair, whereas during the rest of the piece the chairs are carried or 
shoved. There are also some sections in which the microphone is picked up by the performer and also used to 
trigger sounds. These sections are omitted from this article for clarity. 

6A video of Black Vox by Chikashi Miyama: https://www.youtube.com/watch?v=QX l y3v3tk4w (Accessed: 
1/12/2021) 
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Figure 1.2: Empty Chairs by Cathy van Eck (Photo: Marije Baalman, 2019, CC BY) 


the fixed relationships between body movements and sounds in combination with the 
fixed timeline for the change of these mappings the piece can be rehearsed and every 
performance will be pretty similar. 

The relationship between the performer’s movement and the resulting sound—and 
thus between corporeal and imaginary origins—changes too during Empty Chairs (Figure 
1.2). I mainly place three chairs in different positions on stage. In the beginning I position 
the chairs similar to how one would place chairs in a space used for a conference. During 
the piece the chair positions signify other social gatherings: placed close to the audience, 
or all three next to each other at the back of the stage, or two together and one alone—and 
by the end of the piece the three chairs form a circle. These different set-ups of the chairs 
control completely different sonic structures. In the beginning of the piece the sound of 
my footsteps is recorded every time I pick up a chair (the movement of the chair triggers 
the recording to start). When the chair is put on the floor again (and thus stops moving), 
this recording is played back at irregular intervals through the loudspeaker attached to 
the chair. The origins of these footsteps recordings are easily identifiable here, since the 
production has just been heard and seen before. During the piece the movements of the 
chairs control all kinds of different sound processes, starting with sounds close to the 
chairs itself, such as the sounds of chairs being shoved, but soon moving away from that 
recognizable sound by changing the parameters of a polyphonic sampler of more abstract 
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sounds, and ending with acoustic feedback. The corporeal and imaginary sources diverge 
more and more during the piece. In many instrumental pieces the performer’s movements 
can be heard in the sound. The corporeal origins are therefore often mirrored in the 
imaginary origins: fast body movements result in fast-changing sounds, slow movements 
in slow-changing sounds. In contrast to this instrumental practice, the sounds I control in 
Empty Chairs change much more than my actions would suggest. The imaginary origins 
are manifold, compared to what one would expect from the corporeal origins. 


1.3 Composing structures for imaginary origins of sounds 


The interweaving of different kinds of imaginary origins can be found in many of Joanna 
Bailie’s compositions, for example ‘Artificial Environment 5’ from the piece Artificial 
Environments 1-5 (2011).’ As in many of her compositions, Bailie uses field record- 
ings in combination with live playing of instruments in this piece. She describes her 
way of combining field recording with instruments as being based on ‘[...] a kind of 
Cageian/Duchampian belief in the power of framing, the act of transforming real-life 
non-art into art through placing it in an artistic context or by just seeing or hearing it 
in a different way’ (Saunders, 2012). At the beginning of this part I recognize cheerful 
barrel organ music and the noise of happy children’s voices. But then the sound of bells 
starts passing by in a similar rhythm to cars on a road. Since the former movement of 
the piece, ‘Artificial Environment 4’, starts with clearly recognizable recordings of cars 
passing by, my ears are already trained for this sound. I therefore easily recognize the 
car-passing rhythm in the bell sounds. These sounds are accompanied by acoustic in- 
struments, which are often playing downward glissandi, similar to a Doppler effect, and 
therefore enhancing the effect of a car passing by. Pretty soon, though, I not only hear 
bells passing by as if they were cars, but also the barrel organ I heard before, and I seem 
to recognize some car horns. Then the rhythm returns to the barrel organ once more. 

Listening to this mixture of very precisely composed sounds I am in constant doubt 
of what I am listening to. This sonic amalgam can only be created through sounds them- 
selves; I can hear sonic events through structures that seem remnants of other sonic 
events. When listening to this piece, I find myself oscillating between first being able 
to categorize or recognize what I hear, and then not. It would be impossible to compose 
any corporeal origins for these imaginary origins. In my view, these ‘imaginary sound 
sources’ (Chion, 2010, p. 45) would immediately be reduced in their compositional pos- 
sibilities and perceptible interpretations by ‘the visibility of synchronous physical action’ 
(Craenen, 2014, p. 150). To come back to the citation by Paul Craenen at the beginning 
of this text, in this piece, the corporeal origin would diminish the auditory perception and 
impoverish the acoustic fluidity of the sonic materials. 

In In Paradisum, on the contrary, I try to work with these impoverishing qualities of 
the performer’s movements: all kinds of different sounds are connected to one simple 
movement: chewing. Starting with just amplifying the common sounds of chewing an 
apple with every new bite in the apple, other imaginary origins appear. At the beginning 


7A video with score of Artificial Environments 1-5 by Joanna Bailie: https://youtu.be/UWrsqDSL_eU (Ac- 
cessed: 1/12/2021). The piece contains many more elements than described here. An important element is a 
spoken voice, which I left away in my analysis since it is not used in this particular fragment. 
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of eating the apple these are barely audible soft-pitched sounds, but during the perfor- 
mance the apple chews get many different kinds of imaginary origins: they are loud and 
percussive, or long filtered noises. The changes in the corporeal origins are minimal: I 
chew a bit more regularly when triggering the percussive sounds, or I chew more slowly 
when triggering the long filtered noises. One of the main aims of my composition is 
to compose this discrepancy between the movements of the performer and the resulting 
sounds. By creating essentially simple movement-sound relationships (every bite is a 
change in imaginary origin, every chew is a new sound) the changes in this relationship 
become my main focus. These ambiguities between what is heard, what seems to have 
caused the sound, and what is visually happening are all part of composing connections 
between corporeal and imaginary origins. How can the same movement be connected to 
completely different sounds? How musical can chewing an apple become? By compos- 
ing relationships between objects, actions and sounds I try not to answer these kinds of 
questions. I try to raise these doubts through composing the origins of the sounds. 
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#2 

INSTRUMENTAL INFRASTRUCTURE: 
SHEET MATERIALS, GESTURE AND 
MUSICAL PERFORMANCE 


LOUISE DEVENISH 


Thin sheets of material are suspended and placed around the space: tracing paper, silver ac- 
etate and aluminium panels. Smaller aluminium panels rest on low plinths, overlapping but not 
touching one another, surrounded by layers of tracing paper laid beneath them. Before being 
sounded they are still, flat, bright. The flat panels are first activated by rice dropped in single 
grains, and then cascades. The paper is touched with fingertips and nails, swept and rustled, 
before being scrunched and dragged across other material surfaces. The larger aluminium pan- 
els and acetate sheets are suspended vertically, hung like mirrors in two rows. Speakers placed 
behind these hung materials project prepared electronics and acoustic sound. Miniature mi- 
crophones hidden at the performer’s wrists are brought close to the sheets of material, tracing 
lines and shapes on their surface, first simply picking up otherwise inaudible frequencies be- 
fore later being used to induce feedback loops with speakers set behind the hanging layers of 
aluminium and acetate. Sound itself activates the acetate sheets, which tremor and rustle: they 
are sounded by sound. Shards of reflected light are thrown around the space, making sound 
vibrations visible. A performer moves between the corridors created by the hung materials 
that make up this instrumental infrastructure, glimpsed between the mirrored panels as they are 
sounded using hands, mallets, superballs. The flickering light now resembles rippling water as 
the performer moves forwards. Half hidden from view, a single sheet of tracing paper is folded 
into an origami paper boat and placed in the fading pool of rippling light. 


2.1 Introduction 


Over the past 20 years, themes of practice focused on rapidly expanding the tools (in- 
struments, objects, materials), medium (composed music, performed sound) and methods 
(collaborative practice, compositional practice, performance practice) of music creation 


https:/doi.org/10.34727/2022/isbn.978-3-85448-047-1.2. This work is licensed under the 
Creative Commons Attribution-ShareAlike License 4.0 International (CC BY-SA 4.0). 
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and performance have become central to new Western art music. An increasingly sig- 
nificant theme of practice is the musical use of both sonic and non-sonic materials in 
the development of new work. Ciciliani suggests that this is driven by artists who are 
‘working from the understanding that sound alone is no longer sufficient to express their 
musical ideas’ (Ciciliani, 2017, p. 24), yet still seek to root their work in musical practice 
(Walshe, 2016). It is particularly apparent in works with interdisciplinary or intermedial 
influences. As new compositional and performance practices exploring the sonic/non- 
sonic have developed, in some works an entanglement of material roles has taken place. 
In these works, materials that are used instrumentally are also used as infrastructure or 
props, or even as notation.! In a number of percussive works, the distinction between 
instrumental object and implement (or mallet) is also less clear. This entanglement of 
roles is a key feature of an approach I refer to as ‘post-instrumental practice’ .* 

The opening paragraph of this chapter describes Permeating Through the Pores of 
Shifting Planes (2019) by Annie Hui-Hsin Hsieh. Permeating Through the Pores of 
Shifting Planes (hereafter Shifting Planes) was commissioned for Sheets of Sound, a 
percussive performance project by Louise Devenish centred around the use of sculptural 
instrumental installations in combination with electronics.* Although some standardized 
percussive items were permitted, the exploration of non-standard instrumental materials 
was prioritized. The starting point for Shifting Planes included exploration of the in- 
strumentality of various sheet materials, from both a sonic and a non-sonic perspective. 
Three questions guided the development of the work: When using unfamiliar instru- 
mental materials of varying scale, how does the relationship between the visual and the 
auditory change? How can performative gesture be used as compositional material when 
using large instruments distributed around a performance space? How can sheet materials 
fulfil plural roles on stage (instrument, infrastructure, prop, mallet, etc.)? 


2.2 Instrumental sheet materials 


Shifting Planes used a range of sheet materials including paper, silver acetate, and alu- 
minium, as shown in Figure 2.1. The core of the setup was eight custom-made alu- 
minium panels, cut at random in either square or rectangular sections from a 5 mm thick 
aluminium sheet. The four smallest panels were placed horizontally on plinths towards 
the front of the space, with several large sheets of tracing paper placed in layers beneath 
them. Upstage, the four larger panels were suspended vertically. Behind the suspended 
panels were four sheets of acetate, each concealing from view a large speaker on a stand. 


lFor examples of instrumental infrastructure, see Ash and Adam Fure’s The Force of Things: An Opera for 
Objects (2017) or Matthias Schack-Arnott’s Everywhen (2019). For an example of materials used simultane- 
ously as instrument, infrastructure and implement, see Kate Neal’s Never Tilt Your Chair Back (2017). For an 
example of instruments as notation, see Enrique Tomas’ Tangible Scores (2014-2018) or Mark Applebaums’s 
Straightjacket, movement 4: Taquinoid (2009). 

2This forms the basis of my Australian Research Council project ‘The role of post-instrumental practice in 
twenty-first century music’ (2020-2023). 

3Other works commissioned for this project include Percipience: After Kaul (2019) by Louise Devenish and 
Stuart James, and Catacomb Body Double (2019) by Matthias Schack-Arnott. The premiere was presented by 
Tura New Music, with funding support from the Australia Council for the Arts. 
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Figure 2.1: The instrumental setup of Permeating Through the Pores of Shifting 
Planes (Photo: Nik Babic, 2019, CC BY) 


The randomized sizes of the aluminium panels formed a unique and specific pitch set, 
with each panel capable of producing a different range of frequencies depending on how 
and where on their surface or edges they were activated. For example, the largest panel 
produced a clear, low frequency when struck in the centre with a soft mallet, as well as 
very high, sparkling frequencies when struck on the edge with a thin metal beater at a 
90-degree angle. The various materials themselves encouraged different application of 
tools, techniques and movements to each sheet, further diversified by how they were set 
up, which is shown in Table 2.1. In addition to the usual percussive techniques of strik- 
ing, rubbing, shaking, scrunching and dropping, the sonic palette was expanded further 
through the use of microphones and loudspeakers to incorporate non-contact, gestural 
activation techniques. 

The use of microphones and loudspeakers as musical instruments is now a common 
practice, with numerous works using only these materials as their instrumentation. The 
evolution of this practice and associated repertoire has been well documented (van Eck, 
2017), and is common enough within percussive practice that the musical use of mi- 
crophones and loudspeakers has been included in recent pedagogical texts focused on 
percussive implements (Dierstein, Roth, & Ruland 2018). Notably, the use of small 
DPA microphones (such as the 4060 series of miniature omnidirectional microphones) 
attached to performers’ wrists in performance is emerging as a standard 21st-century per- 
cussive technique, as microphones placed at the wrist facilitate close microphone access 
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to sounds and sounding techniques at their source.^ This can be particularly effective in 
works exploring varied proximity to sound, and in works with large or distributed set-ups. 

In Shifting Planes, DPA wrist microphones are initially used to access otherwise in- 
audible acoustic sounds produced by very small movements, such as the tactile sound 
of fingers brushing on paper. In the central section of Shifting Planes the fundamental 
tones of the aluminium plates are sounded using mallets or hands, before an additional, 
otherwise inaudible melodic line is brought forward from the decaying resonance using 
non-tactile gestures to guide the microphone across different areas of the panel to am- 
plify a range of frequencies. This sound world was supported by an electronics part, that 
could be played with or against during this section. As both the wrist microphones and 
the four speakers sounding the electronics are hidden from view, the distinctions between 
who or what is making the sound become blurred. For a short section, the three types 
of sound-related gestures that occur in music performance—sound-producing, sound- 
accompanying or sound-tracing—occur simultaneously (Godøy, 2006, p. 154). Later in 
the work, the wrist microphones are used to stimulate feedback loops with the concealed 
speakers, created by the performance of gentle, specific gestures that are determined in 
rehearsal during tuning of the PA to the performance space. The resonance also activates 
the hanging acetate sheets between them, shown in Figure 2.2. The non-tactile means of 
activation enables the audience to hear the vibrating sheet materials without also hearing 
attack or friction sounds. As the microphones and speakers are concealed and the per- 
former does not make contact with the acetate sheets, again it is not immediately clear 
who or what is making the sounds. Here, a ‘dance of agency’ (Pickering, 2012) between 
human, instrument, microphone and loudspeaker occurs, as the sonic result is not entirely 
controlled by either human or materials. This is further impacted by the acoustic of each 
space in which the work is presented, and is monitored by an off-stage technician during 
performance. 

Godgy hypothesizes that ‘there is a continuous process of mentally tracing sound in 
music perception... mentally tracing the onsets, contours, textures, envelopes, etc., by 
hands, fingers, arms or other’ (Godgy, 2006, p. 149). Hsieh seeks to explore this notion 
in her work, and with regard to the development of In Shifting Planes, states ‘I’m very 
fascinated by the relationship between the visual and the auditory in performance. What 
am I seeing in the performance that informs me about what I’m hearing? How do you 
revert the expectation between you-hit-something, you-hear-something? What happens 
when you give a super big gesture, but the sound is almost inaudible [or vice versa]? 
Is that perceived as loud or soft?’ (Hsieh in Devenish, 2020). With these questions 
in mind, additional compositional materials were drawn from the gestural possibilities 
that emerged from the choices of instrumental materials and how these materials were 
arranged for performance. 

During the creative development of Shifting Planes, it quickly became apparent that 
gestures and techniques that were effective on unamplified materials were ineffective 
when the microphones were used and vice versa. As choreographer Jonathan Burrows 
notes, ‘technique is whatever you need to do, to do whatever you need to do it’ (Burrows, 
2010, p. 68), and for Shifting Planes, this meant drawing on and adapting established 


4In addition to Shifting Planes, some examples of recent Australian percussion works using this technique in- 
clude Matthias Schack-Arnott’s Annica (2016), and a number of the works in Speak Percussion’s Fluorophone 
project (2015). 


GESTURE AS COMPOSITIONAL MATERIAL 15 


Figure 2.2: Non-tactile activation of the silver acetate sheets via microphone and loud- 
speaker feedback loops (Photo: Nik Babic, 2019, CC BY) 


percussive techniques, microphone techniques, performative gestures and everyday ges- 
tures to develop a set of techniques specific to this work, both in the composition and 
performance stages. 


2.3 Gesture as compositional material 


In addition to the materials themselves contributing to the design of the work, a signif- 
icant part of the compositional process included analysis of the performer’s movement 
and physicality used in video and audio recordings of previous performances. In both a 
pre-concert artist talk and an interview, Hsieh described her early compositional process 
as being guided by the natural and musical physical gestures I use in performance: 


Studying gestures tells composers a lot about what [the performers they are writing for] 
prefer to do, how you approach scenarios, what your musical sensibilities are. Follow- 
ing someone’s gesture is a way of understanding your musical choices, the intention 
behind the musical decisions you made, as well as a way of guessing how you move 
and which compositional ideas could work for this performer. How you move is a way 
of showing what you’re thinking. (Hsieh in Devenish, 2020) 


Gesture thus became a source of compositional material in Shifting Planes, providing 
a rich platform for exploration of perception of musical gesture and sound. The use of 
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Instrument Implement Activation method 


Dropped in single grains, and in small handfuls 


Rice and small plastic beads Poured from a bottle in cascades 
Aluminium Bounced off a drumhead 
panel Mallets: Struck 
Bass drum beater Struck 
Yarn mallet Struck (tips and shafts) 
Dreadlock mallet dragged (surface and edge) 
Superball Rubbed across reverse side of plates 
Fist 
Hands Fingertips 
Nails 
Microphone feedback, electronic sound | Activated by vibrations from the loudspeakers 
Hands, nails, fingertips Scrunched, folded, brushed 
Tracing paper Rice and small plastic beads Bounced off the bell plates, brushed 
Dreadlock mallet Dragged, caught 


Microphone feedback, electronic sound | Activated by vibrations from the loudspeakers 
Hands Shaken gently 


Acetate Sheets 


Table 2.1: Activation techniques used on the sheet materials in Shifting Planes 


physical gesture as compositional material has been a cornerstone of percussion music 
for decades, cemented by the large body of instrumental theatre works composed for 
Trio Le Cercle in the 1970s and 1980s by composers including Mauricio Kagel, Georges 
Apherghis and Vinko Globokar. In recent decades, composers such as Jessie Marino, 
Natacha Diels and Jennifer Walshe have continued the trajectory of gesture as compo- 
sitional material. Over time, the three primary methods for employing gesture in per- 
cussive music that have emerged are choreographic, tactile and virtual. Choreographic 
gesture does not directly contribute to sound making, in that it does not produce or cap- 
ture sound, nor does it connect with any instrument other than the body. For example, 
Jessie Marino’s trio The Flower Episode (n.d.) is a rhythmic group choreography of 
six hands, with most gestures not requiring the intentional production of sound. Simi- 
larly, Mark Applebaum’s Aphasia (2010) and Ceci n’est pas une balle (2012) by Mathieu 
Benigno, Alexandre Esperet and Antoine Noyer require the performer to silently execute 
specific, precise gestures; however, in these works this is done in time with a pre-recorded 
tape. Aphasia is described by the composer as ‘essentially a choreographed dance work’ 
(Applebaum score 2010), with the gesture aligning with pre-produced sound, rather than 
generating live acoustic sound. In contrast, tactile percussive gesture is used to pro- 
duce sound through connection with acoustic objects or instruments, or to capture sound 
from activated instruments using microphones, and these approaches are frequently used 
together. For example, in Juliana Hodkinson’s Lightness (2015, rev. 2018), three percus- 
sionists strike matchsticks on ignition ‘runways’, rhythmically drag matchsticks across 
different grades of sandpaper, and precisely extinguish lit matches in small trays of water. 
The quiet, delicate, or proximal sounds that result from these gestures are captured by 
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wrist microphones worn as described above. Finally, virtual gesture is common in the 
performance of MIDI instruments or software programs which rely on sensors to capture 
specific performative gestures in space to activate electronic sounds or to trigger samples, 
such as Jean Geoffroy and Thierry de Mey’s Light Wall System. 

Gødoy observes that gesture-based control of digital musical instruments (DMIs) can 
be seen as an evolution of musicians’ relationships to traditional acoustic instruments 
(Gødoy, 2006), and the roots of these techniques can be connected to standardized in- 
strumental techniques. Although all of the works described above are frequently per- 
formed by percussionists, any musician or actor could perform these works. Marino 
refers to the kind of performance skills required for her work as a ‘virtuosity of the ev- 
eryday' (Marino, 2020), using performance techniques that foreground everyday human 
gestures. In Shifting Planes, everyday gestures such as pouring from a bottle, brush- 
ing materials off surfaces, standing before a mirror, and folding a simple origami paper 
boat are woven together with percussive and microphone gestures. This kind of blended 
performance practice requires another type of virtuosity common in new music, which I 
have previously referred to as a ‘new virtuosity’ (Hope & Devenish, 2020). 


2.4 Instrumental infrastructure and notation 


In Shifting Planes, the multidimensional role of the sheet materials emerges through their 
performance. The sheet materials used in this work first appear as a theatrical set, within 
which a performance will take place. As the work progresses, the materials on stage are 
transformed (via their use in performance) from stage infrastructure to musical instru- 
ment and back again. The varied roles of the instrumental materials and their engagement 
in performance is notated using a hand-drawn action score that included a combination 
of text instructions, pictographic representations of physical gestures, and limited ele- 
ments of conventional music notation. Shifting Planes is scored on graph paper, with 
each unit representing five seconds, and is relatively sparsely notated. An excerpt of this 
score is shown in Figure 2.3. Hsieh believes that the composer is never really in control 
of physical gesture, and that ‘the more important the gesture, or the more featured it is in 
performance, the less it should be notated' (Hsieh in Devenish, 2020), as everyone has a 
unique body and unique ways of moving. 

Each of the three sections of the work comprise notations addressing gestures and 
actions that the performer should execute. The first section is entirely text based, with 
written instructions such as ‘slowly tip out’, ‘gently swirl the remaining grains’ and ‘lis- 
ten and react to the texture heard in the fixed media'. The second section introduces 
line-based pictographic notation, which depicts gestures to be ‘drawn’ on the surface of 
various sheet materials using a range of indeterminate implements. Line-based gestures 
and their position on a surface or in space can be easily notated, either on standard- 
ized five-line manuscript or using graphic notation. It is particularly useful for music 
not bound by pulse or metre—that is, led by listening, movement and play. The act of 
drawing or tracing lines or outlines on surfaces or in space as a means to elicit sound ap- 
pears in a range of ways in the percussive repertoire. For example, lines can be executed 
rhythmically using short rapidly drawn lines, as in my realization of Mark Applebaum's 
Composition Machine No. 1 (2014) using a thick texta pen on amplified butcher's paper. 
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Alternatively, they can be drawn texturally using large circles drawn using a slow, smooth 
motion, as in my realization of Cat Hope’s Tone Being (2016) using cardboard tubes on 
a tam-tam (Devenish, 2018). In the former, the lines leave a trace on the butcher's paper, 
resulting in visible graphic drawings. In Hope’s work, and in Shifting Planes, lines are 
drawn using various percussive implements and do not leave a visible trace. 
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Figure 2.3: Excerpt of the score Permeating Through the Pores of Shifting Planes 
showing hybrid notation (Photo: OA. Hui-Hsin Hsieh, 2019, used with permission). 
Dynamic graphic score created with Decibel Score application (2013) 


2.5 Conclusion 


The increasing entanglement of materials as instruments, infrastructure, implements and 
increasing diversification of performance technique is effectively breaking down bound- 
aries between musical practices: composition, performance, music, movement, instal- 
lation. The design, layout and use of tangible and intangible instrumental materials is 
increasingly influenced by how performers might interact with these materials using mu- 
sical, performative and everyday gestures and techniques and vice versa. Permeating 
Through the Pores of Shifting Planes was an initial exploration into the sonic and perfor- 
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mative affordances of sheet materials, with particular consideration given to the relation- 
ship between the visual and the auditory in performance. The detailed investigations into 
gestures, a dispersed, installation-like setup, and individual performative movements at 
the early stages of development allowed almost all of the original ideas to be realized in 
performance. The use of my gestural, material and electronic instrumentarium was an 
effective way to ‘grease the wheels’ of a new collaboration, making use of all of the cre- 
ative languages at our disposal. New music premieres always seem to take place with less 
time in the venue on the complete set-up than is ideal, and Hsieh’s consideration of my 
performance style, interests in improvisation and graphic notations as well as her con- 
sideration of familiar percussive instruments and techniques in the early stages, resulted 
in an idiomatic work that came together quickly in the final stage and rehearsal period. 
Furthermore, this allowed deep and layered exploration of the questions that drove the 
work throughout the entire process. 

The research questions surrounding visual-auditory relationships in performance, sheet 
materials as instruments and the plurality of material roles in performance, were explored 
on micro and macro levels over the course of Shifting Planes, using large gestures to ex- 
pose small sounds, and small gestures that (with the assistance of amplification) conveyed 
large sounds. The notion of large-small and small-large gestural/sonic combinations and 
consideration of how the visual can affect the auditory have appeared in percussive prac- 
tice and pedagogy for decades. For example, young orchestral players are encouraged 
to ‘think loud, play soft’ to achieve the perfect pianissimo triangle note, and to con- 
sider how much more effective a crescendo shake roll on the tambourine seems when 
the shake roll is accompanied by a gesture that raises the tambourine from low to high. 
Similar gestural concepts are embedded in all percussive techniques. When combined 
with the wrist microphones, these concepts can be explored further, particularly when 
working with feedback loops. When there is no attack to rebound from and the sound 
emerges from the aforementioned ‘dance of agency’, the performer can focus their ges- 
ture wholly on the care of the sound they are working with, and the ways in which gesture 
can guide listeners. In Shifting Planes, these concepts saw the transformation of an inani- 
mate instrumental ‘set’ that a performer sounds, into an animated, vibrating instrumental 
infrastructure that a performer responds to, and this has formed the foundation of further 
collaborative artistic research. 
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#3 

PRE-GESTURE, GESTURE AND SOUND 
ON [NO] PIANO: 

MUSIC FROM SOMEWHERE 


HAIZE LIZARAZU 


...real space is neither its topos nor its instrument. This music is not here or there; as soon as it 
sounds, it is all-present and all-penetrating [...]. Therefore let us call this music, from a spatial 
perspective, a music from somewhere... 

—(Craenen, 2014, p. 30) 


3.1 Introduction 


Let this initial quote be the starting point to talk about Music from Somewhere, a piece 
written in 2017 (reviewed in 2019) by the composer Fran MM Cabeza de Vaca.! Mu- 
sic from Somewhere, for pianist’s hands, lights and stereo audio tape, shows us a simple 
scenic display to amplify the previous moment to the sound emission of the piano, work- 
ing with the silence as material that precedes the music that is yet to come, the one that 
is in this other place. 

Bringing that specific moment to the forefront means that the movement and gesture 
become our main visual input, focusing on that virtual place where the performer is just 
before the sound materializes. The performer explores the different ways to connect the 
body to the instrument, linking it not only to the visible gestures, but also to the previous 
state where the invisible movement stays: the pre-gesture. 

My background as a performer started with a classical piano education in a music 
conservatoire. Later on, I specialized in contemporary music, combining it with specific 
interest in free improvisation and sound art.” As the performer of this piece, I found that 


'The video documentation of the work is accessible here: https://vimeo.com/449604575 
?To learn more: https://www.haizelizarazu.com/en/ 


https:/doi.org/10.34727/2022/isbn.978-3-85448-047-]_3. This work is licensed under the 
Creative Commons Attribution-ShareAlike License 4.0 International (CC BY-SA 4.0). 
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Figure 3.1: Music from Somewhere by Fran MM Cabeza de Vaca (Video ©Marta 
Azparren, 2020) 


Music from Somewhere posed some questions from the very beginning of its conception: 
How do I relate myself to the piano when it is not physically present? How are my 
pianistic movements linked to the score when the resulting sound is not a consequence of 
the notation? All these lines converge in the self-consciousness of my moving body, on 
the proprioceptive process of the pianistic gestures that I have acquired through all my 
student years. Music from Somewhere challenges the pianist to rethink all the mechanic 
and automatized movements, the visible and the invisible, the different kinds of memories 
we develop, and creates a new space where sound, silence, gesture and pre-gesture are 
inter-connected. 


3.2 The invisible: score and pre-gesture 


There are many different kinds of scores, ranging from traditional staff notation systems 
to graphical scores, to video- or aural-scores. All of them are related to sound and show 
us various ways of representing it. When digging into other artistic disciplines, however, 
such as dance, theatre or performance, we can also find the score concept—not related 
to sound, but to movement and/or actions. Richard Schechner (2002) links the score to 
what he terms ‘proto-performance’ (p. 234). He describes the proto-performance (or 
‘proto-p’) as something that precedes and/or gives rise to a performance: a starting point 
or, more commonly, a bunch of starting points (Schechner, 2002, p. 225). 

The score in Music from Somewhere is twofold in this way: it is a video-score that is 
synced with the music (tape) that uses—mainly—two different kind of notations: ges- 
tural/positional (Figure 3.2) and traditional musical notation (Figure 3.3). The former 
clearly asks the performer for an imitation kind of response: do what you see. The latter, 
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Figure 3.3: Traditional musical notation 
on Music from Somewhere (Score ex- 
cerpt) 


Figure 3.2: Gestural notation on Music 
from Somewhere (Score excerpt) 


apparently essentially traditional piano notation, asks the performer for a very different 
type of work: they need to find a gestural approach to the given piano score excerpts. 
Many of them belong to pieces that I have, at some point in my life, already studied 
and played; for example, works from the traditional, well-known piano repertoire, such 
as Beethoven’s piano sonatas, Webern’s op.27 Variations or Debussy’s Études (among 
others). All these different pieces are therefore linked to my own experience. To my 
own memory. Of course, we can’t talk about memory as a single concept, as it has many 
different layers: aural, tactile, emotional, analytical, muscular... but all of these layers 
converge at one common point: the body. As Paul Craenen (2014) explains, 


besides being a perceptual filter that determines what can be perceived and experienced, 
the body also functions as an active and reactive memory. Mimetic theories have already 
made us aware of the close relationship between action and perception, the ability to 
perceive, and muscle memory. (p. 265) 


Furthermore, it is very important to mention that the score excerpts do not try to rep- 
resent the resulting sound, as traditional scores do, but rather set a trigger to the action 
of approaching the written sound. Or, to put it another way, they explore the silence that 
exists just before the sound is produced (the music of the piece is in a great part a compo- 
sition of the many different silences of many different piano performances). Hence, no 
actual piano performance gesture is desired in the first half of the piece. Taking all these 
considerations into account, performing the aforementioned score sketches lead me to 
re-visit them, remember them in all the possible ways: aurally, gesturally, emotionally. 
During the performance of the piece, no piano is there to respond with a resulting sound, 
or to feel its physical resistance. Therefore, the study process goes back to the real piano; 
not to practising the score itself, but to practising the feeling, the perception of playing 
those different piano scores. Very little time is given to change from one score to another, 
from one style to another, from one memory to another. Hence, the focus of the practice 
process relies on the awareness of your own body, on the previous state of playing the 
notated sound, on what I call the pre-gesture. 
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The pre-gesture is the invisible gesture that precedes the visible one. It is a start- 
ing point, an impulse. Much like the definition that Schechner makes about the proto- 
performance. It is very interesting to observe, too, how in this same line, the author 
gives a special emphasis to the concept of pre-expressivity (Schechner, 2002, p. 226), a 
term related to the theatre author and director Eugenio Barba (previously developed by 
his teacher, Jerzy Grotowski).? It all makes reference to this previous body state—body 
presence—that the performer needs to train in order to actually perform the play, text, 
work or dance. 

The same way that a performer/actor trains the presence of their own body, the ten- 
sions preceding the actions (the in-tension) in order to acquire an incorporated knowledge 
through the body, the performer/pianist in Music from Somewhere needs to dig until the 
pre-gesture level, the preceding action, the previous mind-body state, in order to perform 
the pianistic gestures without either a piano or a resulting sound. Figure 3.4 shows how I 
recorded myself playing the different score excerpts; I used this as a helping tool to mem- 
orize the visual gesture as seen from the outside (added to the proprioceptive feeling of 
the pre-gesture). 


Figure 3.4: Music from Somewhere: screenshots of the video documentation during 
the study process (Photo: Lizarazu, 2020, CC BY) 


All these concepts (proto-performance, pre-gesture, pre-expressivity) circle around 
the same point. On the subject of this previous state of the body Paulo de Assis (2018) 
provides a very interesting analysis about Simondon's key concept of transduction and 
how the body could be understood as a transducer in music-making (pp. 137-157). 
Terms such as structural germ or micro-haecceity (related to the Simondonian notion of 
individuation) illuminate these energy and state changes of the performing body and give 
us a clue about how we might apply it to the musical realm. In his words, 


3To learn more, see Barba, Eugenio & Savarese, Nicola. A Dictionary of Theater Anthropology: the secret art 
of the performer. Routledge, New York (1991) 


THE VISIBLE: GESTURE AND SPACE 25 


Micro-haecceities are high energy-loaded and high-speed-moving singularities that carry 
a force of potential from one position to the next. They make up the visible or audible 
part of artistic transductive processes. In their functioning as radical becoming they 
never appear as stable beings, remaining an impulse of virtuality from one actualisa- 
tion to the next. If one thinks, or does, or experiences artistic performances with these 
operations in mind, the Deleuzian notion of capture of forces becomes more graspable 
than ever: the virtual becomes actual in order to be instantly dissolved into the virtual 
again. The pianist playing Schumann perfectly exemplifies such a capture: he or she is 
not merely reproducing a stratified, pre-existing entity, but operating a capture of forces 
(from the virtual) that produces a new individuation (actual) as a highly intensive be- 
coming, which immediately—as soon as it is generated—points forward to other virtual 
pre- and after-individualities. (de Assis, 2018, p. 149) 


As we see, the continuous contingencies that happen within a performance (virtualities) 
are a key concept in understanding how our body moves and behaves in the music-making 
process. The author refers to the structural germ as a potentiality: it is a structural consec- 
utive potential; that is, it carries some sort of information, which sets the basic conditions 
for an event to happen (de Assis, 2018, p. 147). It is clear now how all the mentioned 
terms and theories help to facilitate an understanding of the body and its movements not 
only in the performance moment, but also in the previous non-visible states. Music from 
Somewhere explores all these pianistic pre-gestures as the main musical and working 
material. It challenges us to focus and to be conscious of them not only on a concep- 
tual level, but also as an interaction process between the body and the instrument at the 
practical level. 


3.3 The visible: gesture and space 


At first sight, when watching Music from Somewhere, one can see that two main spaces 
are created and shown: the public space in front of the white panel, and the semiprivate 
space in the background, where a dim light and a distant music carries us. The pub- 
lic space is presented as the performing stage, where the hands, fingers and arms are 
shown. A kind of concert stage. The semiprivate space is presented as an intimate study 
place (home, studio), where the performer usually finds themself alone (even though the 
audience is able to ‘peep’ them through the panel hole). The proposition, here, is one 
of outside and inside; what the audience does or does not see. And between those two 
places, once again, a common agent: the performing body. 

These two spaces have different purposes. On the one hand, it serves as a visual and 
clear separation for and between the audience and the performer, who hides behind the 
white panel. On the other hand, it physically creates the different places where sound 
exists in a performance, as we will elaborate in the following lines (the moment just 
before the sound, when it sounds, after the sound is heard/played). This second partition 
leads us to talk about the space in aural terms: the sound-space. 

The first case scenario is the sound-space that happens before the note is played, before 
the hand actually touches the keyboard. This starting point (which itself consists of many 
starting points) is linked with with the aforementioned pre-gesture, as we have seen. But 
that pre-gesture is not just a physical proprioceptive feeling, it contains all of the music 
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that is yet to come. To be at the pre-gesture level, the performer needs to be also at the 
multiple levels and ‘places’ that the piece has been before, since the first reading until 
the performance moment. For Mine Dogantan-Dack (2011), ‘the unity of the initiatory 
gesture and tone produced is also part of the listening experience, although the listener 
is not ahead of the music physically in the same way as the performer is’ (p. 259). The 
audience doesn’t just listen to the sound of the piano, or the various attacks of the pianist. 
The audience sees how the performer moves, how they approach every musical event, 
and also, how they enter the stage or enact in certain, idiosyncratic ways, the music that 
we're about to experience. As Dogantan-Dack (2011) discerningly points out, ‘in fact, if 
we really are precise, it is not the attack that produces the sound, but the gesture bringing 
about the attack’ (p. 259). 

It should now be clear for the reader how paramount this previous state of the per- 
forming body is—not only for the performer themself, but also for the audience (that 
perceives it from a third person perspective) and for the resulting sound that comes after. 
As Dogantan-Dack (2011) explains: 


(...) the performer starts to experience the tone much earlier not only mentally, but 
also physically, at the beginning of the fixating gesture, before the hammer contacts the 
string and the tone actually starts sounding. The kinaesthetic sensations that accompany 
the gesture result from the adjustment in muscular tonus that the pianist makes to pre- 
pare the impact, and this adjustment in turn is guided by an aural image of the desired 
tone, the goal of the gestural movement. Pianists know — must know — what kind of 
tone will ensure that their touch is able to produce the intended tone at all. (p. 258) 


The sound-space concept has been examined for many decades now, from the archi- 
tectural perspective as well as from both the listener’s and a music-making perspective. 
Often sound is introduced in this discourse as an isolated element, with the argument 
being that sound creates the space itself, like a sphere without fixed boundaries, space 
made by the thing itself, not space containing a thing (LaBelle 2010, xxi-xxii). Though 
this argument may be true in context, we might also think about the impact of the music- 
making body in the sound-space creation. It is important to think about spatiality not 
only in physical, three-dimensional terms, but also as a musical metaphor to understand 
the different moments and events that occur when the sounding body intervenes. Paul 
Craenen (2014) provides an extensive analysis on this subject, in which he traces music 
as a spatial phenomenon through different categories, such as: a) space surrounding the 
music; b) space for the music; c) space of the music (pp. 20-25). 

The first category refers mainly to the characteristics that make musical activity rec- 
ognizable in a society, meaning the role that music plays in our social, cultural, political 
and economic context. Here, the human body is found in a multiple layer scenario where 
audience, politicians, promoters, programmers, musicians all interact in creating this 
surrounding of the music as a metaphorical category (Craenen, 2014, pp. 20-21). The 
second sound-space is related to the instrumental space. This is where we find the music 
halls, venues, stages and all the specifics needed for the music performance to happen. It 
is also where the performing body first appears as a main character. Lastly, the space of 
music itself defines what we understand as the musical playing field. In the words of the 
author, 
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...the musical playing field (...) is a space that is constantly shifting, a space that 
behaves dynamically. It is an emerging space that unfolds as the music progresses. The 
musical playing field is a cultural and audio-motor space in which patterns of musical 
expectation, stylistic characteristics, and idiomatic sonorities encourage selection and 
variation. (Craenen, 2014, p. 23) 


The space of music changes from piece to piece, from performer to performer, from 
instrument to instrument, even if the space for the music remains the same in all those 
changes. It is in this musical playing field where the listening and perceiving experiences 
carry us to other, more intimate sound-space metaphors: the music from here, music 
from there and music from somewhere (Craenen, 2014, pp. 26-30). Focusing on this last 
metaphor (which gives its name to and also partly motivated the piece we are analysing 
in this paper), this somewhere attempts to name that space where the music carries us, we 
listeners and performers, when sounding,. This place is formed not only by the sound, 
but also by the movement, the performing body that materializes that sound. 

In Music from Somewhere, all these sound-space metaphors are presented through 
the gesture of the performer. First, the sound is being—wanting to be—materialized, 
present: here. As we have already seen, this is managed through the inner perception of 
the pre-gesture level. Secondly, once the performer’s finger literally touches the table— 
the [no] piano—another sound-space is presented, the one where the sound is played 
and long gone at the same time: there. In that place, the visible pianistic gestures are 
more recognizable than ever. The attack, the intensity, the tension, the release of the 
notes is being shown as a consequence of the proprioceptive memory of the performer 
and their embodiment. The travel through these two places lead us, the performer and 
the audience, to this other place: somewhere. From a spatial perspective, a music from 
somewhere is conceived as a fluid, phantasmal space experienced kinetically (Craenen, 
2014, p. 30). 

The awareness of the different sound-spaces in relation to the present body (perform- 
ers and audience) enables us to analyse the performing body, its gestures and movements 
as an independent and well established concept for theorization. It is in fact the co- 
presence of the bodies that makes the scenic realization possible and the one that set the 
ground for the first performative turns in the arts (Fischer-Lichte, 2004, p. 65). The 
importance of the gesture in relation to a musical instrument is a key component in the 
learning process of any musical piece at any musical level. 


3.4 Conclusion 


Music from Somewhere is a musical piece that uses the gesture and the performing body 
as musical material. It paradoxically explores the relation between the pianist and their 
instrument by eliminating the latter. This sets the focus on the movements that a pianist 
does not only when performing, but also when practising. The two main points that 
have been explored through this piece are the self-awareness of the pre-gesture and the 
creation of the sound-space through the gestures of the performing body. 

Pre-gesture has been presented as an inner movement of the performer, a previous state 
of the performing body where all the music, information and possibilities live before the 
sound actually materializes. The concept is here connected to other artistic fields, such as 
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theatre and performance, where the body has been a more extensive subject of study than 
in the musical field (Schechner, 2002; Barba & Savarese, 1991; Lecoq, 2003). I argue 
about the importance of focusing on this previous state to understand the embodiment 
of pianistic gestures and the cognitive processes enacted in the study and practice of a 
musical piece or instrument. 

In addition, pre-gesture also serves as a practical tool to perform and elaborate that 
metaphorical place where the music exists prior to its sounding. The actual materializa- 
tion of the sound leads us, then, to this other sound-space, where sound exists and is gone 
at the same time, a fleeting moment where sound is barely localized. And in all of those 
sound-spaces, the performing body acts as an interface, a filter, a bridge to carry us to 
that somewhere, where all the music, the audience, the performers exist in a non-spatial 
space realm. 
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EVOLVING MUSICAL EXPECTATIONS: 
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HUMAN MUSICIAN AND AN Al MUSICAL 
INSTRUMENT 


KORAY TAHIROGLU 


4.1 Introduction 


Artificial intelligence is one of the most active areas of research in the music technol- 
ogy community. There are significant efforts to further our understanding of artificial 
intelligence’s potential applications in music on behalf of many practitioners, artists, 
musicians, computer scientists, with their own particular approaches, goals and their 
contributions to the development of AI research in music. One of the main areas of focus 
has increasingly been on expanding the current use of artificial intelligence to the creative 
practice of musicians, making new technologies available and accessible to instrument 
builders, musicians and composers (Eigenfeldt & Kapur, 2008; Tatar & Pasquier, 2019). 
Currently, there is an ever-growing demand for tools and techniques that allow for the 
creation of autonomous devices and processes (Tahiroglu, Kastemaa & Koli, 2020). Fol- 
lowing the demands of musicians, a considerable amount of research has been devoted to 
various approaches to manipulating and transforming musical instruments into musical 
agencies through the autonomous acts of the musical instrument in a collaborative mu- 
sic action between musician and the musical instrument (Tanaka, 2006; Karlsen, 2011). 
Such musical instruments can incorporate features that allow them to act in mutual co- 
operation with human musicians in the process of composing and performing music. In 
this joint activity, the development of algorithms that can contribute to compositional 
structures as well as composition methods is a promising path towards the integration of 
more advanced computational technology with the creative practice of musicians. 

In this article, I intend to reflect on the potential use of artificial intelligence tech- 
nologies in instrument building, and subsequently to address the specific challenges and 
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opportunities that arise through these technologies. I focus on emerging musical ex- 
pectations and musical demands in a practical research implementation of our AI-terity 
autonomous musical instrument. Relating advanced technology to music and perfor- 
mance is not a new phenomenon and has been discussed in more depth (Collins, 2007; 
McPherson & Tahiroglu, 2020; Magnusson, 2019; Tahiroglu, 2021), at the same time, 
not much effort has been spent in examining the following questions: How do artificial 
intelligence autonomous algorithms and human musicians contribute to new musical ex- 
pectations? What can be realised through the equal contribution of AI instrument and 
human musician in a music performance? 


Figure 4.1: Al-terity instrument (Photo: Koray Tahiroğlu, 2020, CC BY) 


4.2 Al-terity and the composition Uncertainty Etude #2 


In our Sound and Physical Interaction (SOPI) research group at the Aalto University 
School of ARTS we have been building, developing and performing with the AI-terity 
musical instrument (Figure 4.1), which is a non-rigid instrument that comprises a deep 
learning model, GANSpaceSynth,! for generating audio samples for real-time audio syn- 
thesis (Tahiroglu, Miranda & Koli, 2020). Physical deformability becomes the affor- 


! GANSpaceSynth is a hybrid generative adversarial network (GAN) architecture that we developed in our 
SOPI research group. It applies the computational features of the GANSpace method (Härkönen et al. 2020) 
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dances of the instruments for handheld physical actions to be applied. The instrument 
uses an abstract form of an interface that is responsive to manipulation through bending 
or twisting, controlling parameter changes in granular synthesis. 

We developed the GANSpaceSynth, specifically to provide more control over the spa- 
tial exploration of the audio features that are distributed in the latent space. This unique 
access to the GAN latent space gives musicians the ability to interact with the higher 
order structures to generate new audio samples. The advantage of this approach is the 
control of the directions for moving one point to another in spatial dimensions of the 
latent space. We call this point, the synthesis centre point (SCP). 

Following the unique features of the instrument’s deep learning model, we developed 
the instrument’s autonomous features further to bring in alternative musical responses in 
music performance (Tahiroglu, Miranda & Koli, 2021). In this way we could approach 
the performance of a music composition as an entity providing an independent vari- 
able that could affect the musical context by changing the decision with a non-arbitrary 
way of generating new sounds. To do that, we built in autonomous features to change 
the direction of the SCP in the latent space. GANSpaceSynth generates audio samples 
based on the SCPs, and these points determine the audio characteristics of the samples. 
The musician can navigate through the latent space by interacting with different parts 
of the interface, and GANSpaceSynth receives each of the SCPs as input to generate a 
corresponding audio sample. The idea of the autonomous behaviour is to monitor the 
musician’s state of performing with the instrument and change that confident state of 
performing to an intermittent state of performing. The autonomous behaviour allows for 
alternative sound-producing expressions to appear; these are then layered in the changing 
audio features of the real-time granular synthesis. In this way, the autonomous nature of 
the instrument can be seen as an autonomous behaviour that aims at keeping the musi- 
cian in an active and uncertain state of exploration, which allows massive flexibility and 
instantaneous exploration of an instrument’s playability. 

We wrote a composition for the AI-terity instrument, aiming to idiomatically reflect 
the autonomous features of the instrument. The work is based on the idea of uncertainty, 
where the instrument moves the SCP across latent space, aiming to find a new target 
point, but never stays in one particular point long enough to allow the musician to stay 
in a comfortable and certain state of performing. The composition brings up some con- 
fusion and surprise. Figure 4.2 shows the studio recording of the piece.” I should clarify 
precisely what is meant by ‘uncertainty’ in this case. It doesn’t mean that the instrument 
chooses any ‘random’ points in the latent space; the jumps in between the SCP that the 
GANSpaceSynth uses to generate audio samples for the granular synthesis are not ran- 
dom. Instead, the autonomous features are designed to move in the opposite directions 
on the basis of the latent space centre and gradually introduce new audio samples in a 
smooth transition that allows the musician to explore the changing timbre of the audio 
samples during each jump. Through this process the music is composed—and it is un- 
predictable. You can hear it changing all the time. For a while follows the course of 
playing in an original set of generated audio samples, but the music is composed for each 


on the audio synthesis features of the GANSynth model (Engel et. al. 2019) for organising the latent space 
using a dimensionality reduction for real-time audio synthesis. The open source code of the GANSpaceSynth 
is available at https://github.com/SopiMlab/GANSpaceS ynth/. 

>The studio recording is available at https://vimeo.com/5 14201580 
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jump with the intention of bringing in a new set of audio samples. It is possible to create 
a certain combination of sounds, each combination being a result of the SCP. What the 
performance of the piece attempts to do is to create unpredictable yet original musical 
expectations for the musician and the audience. 


Figure 4.2: Studio recording of the composition Uncertainty Etude #2 (Photo: Koray 
Tahiroglu, 2019, CC BY) 


4.3 Achieving musical expectations 


Titon (2009) describes a model for our interaction with music that presents musical per- 
formance as the process of creating the active experience of musical works. Following 
this model, we can see that music performance makes both those musical and aesthetic 
expectations appear that are already present before the act of the performance, as well as 
those that are not. In the context of music with an AI powered autonomous instrument, 
musical expectations appear in relation to the mutual connections between the instrument 
and the human musician. The performance of the piece Uncertainty Etude #2 explores 
how the artificial intelligence instrument can serve as a musical partner, so that the human 
musician and the instrument can communicate to evolve musical expectations. 

Musical expectations have been discussed in the context of melodic (Margulis, 2005), 
tonal (Bharucha, 1994), sound and meaning (Clarke & Cook, 2005), biologically learned 
(Huron and Margulis, 2010), memory and cultural (Curtis and Bharucha, 2009) patterns 
involved in music that may give rise to expectations and to 'affective consequences of 
expectations' (Huron & Margulis, 2010) for listeners or musicians. Huron and Margulis 
(2010) mention that ‘familiar experience’ occurs in the nature of musical expectation 
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even between songs devoid of any structural musical relationships. It is an indication 
that familiarity could play a role in helping listeners to learn how to anticipate future 
songs, an ability which is often attributed rather to the listener’s “general knowledge’ 
of the music. This expectation, however, is not a direct knowledge, and only shows to 
the listener the potential that is inherent in the music. By contrast, expectations have 
a proprietary quality, in that they can be learned from the specific events themselves. 
Regarding the indications for ‘anticipating the future sounds’, the question then arises, in 
what ways can we expect a particular musical expectation to be achieved in a correlation 
of sounds that even the musician has not heard before performing the music? For the 
composition Uncertainty Etude #2, we trained the GANSpaceSynth model with the audio 
dataset that has the overall textures of musical sounds, essentially inharmonic and atonal 
features with electronic ambience patterns. I have provided my own dataset for training 
the GANSpaceSynth checkpoint. The resulting checkpoint model in the composition 
tends to generate rather garbled approximations of the original dataset with smeared 
transients of unfamiliar sounds that do not follow the musical features of tonality or 
the context of melody as commonly discussed in the forms of musical expectations. 

In the performance of the Uncertainty Etude #2, it might be argued that the ability to 
form and achieve expectations about unfamiliar sounds could still be more innate, while 
at the same time to some extent being dependent upon musical experience. This leads 
us to discuss whether musical expectation could be formed and developed on the basis 
of some other behavioural responses, such as intersubjective experience (Fuchs & De 
Jaegher, 2009), that is, from hearing other listeners. Perhaps it could be learned; indeed, 
as the structure of the musical language in Uncertainty Etude #2 changes, some of the 
features required to form expectations are then learned through the performance of the 
piece. It might not be too much to argue that achieving musical expectations about the 
performance of the Uncertainty Etude #2 will occur through direct acquaintance with 
the music. The mutual correlation between the human musician and the autonomous AI- 
terity musical instrument will evolve unfamiliar and surprising musical expectations that 
the listeners will experience. 

Bharucha (1990) questions whether the listeners or musicians give up the element of 
surprise in musical expectation when they prioritize what they have already known. Iam 
not sure if there is any particular answer to that question, but it might still be worth men- 
tioning that familiarity has the potential to provide an emotional boost to music listeners’ 
enjoyment, which might counteract the effect of surprise in music enjoyment. If this is 
true, then, in contrast to the music enjoyment of listeners of unfamiliar genres, are the 
listeners of familiar genres more likely to experience surprise as an unexpected loss? 

It could be further questioned whether unfamiliar genre-specific audiences have a 
greater expectation of novelty than listeners of familiar genres. I think this may be more 
plausible than it would be in the case of other kinds of anticipation and expectation; but 
even if the question is theoretically open, it would be a tough argument for the listener 
of a familiar genre to come to the conclusion that any song is more interesting than, say, 
a song that fits the familiar genre. In the performance of the composition Uncertainty 
Etude #2, there is a unique set of properties for musical expectations, which is the result 
of the mutual cooperation between the human musician and the autonomous instrument. 
This set of properties are part of the unfamiliar and surprise nature of the musical perfor- 
mance itself. 
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4.4 Musical expectations as a framework for composition 


The shift of music performance from a process with a master performer to an activity 
performed by a human musician and an autonomous instrument can also be considered 
as a particular social expectation that can be used as a framework for a musical com- 
position (Tahiroglu, 2021). In the performance of the piece Uncertainty Etude #2 the 
human musician and the instrument become part of a collective, part of a “performance 
ensemble’ (Latour, 2005). There is a ‘particularity’ in this collective in its own way. The 
particularity, here, can be expressed in its musical and conceptual form, which can be de- 
rived from the composition. As part of the performance ensemble, the human musician 
and the instrument become a ‘system of instruments’ with their own musical demands, 
and yet also a ‘system of musical compositions’. 

Such performance ensemble opens a space for ways of music-making in which unfa- 
miliar and surprise musical expectations become a part of compositional structures. We 
can consider this process as a compositional framework that offers insight into the cre- 
ation of compositions that are fully or partly autonomous, or autonomous in an unusual 
way. We can speak of unfamiliar and surprise musical expectations with an immedi- 
acy that the performance would entail. There is a potential for musical exploration that 
arises from particular musical expectations, in which the musical structure and its mate- 
rial have to do with something that autonomously evolves with unexpected, surprise and 
unpredictable musical events. 


4.5 Conclusion 


The intention of this chapter was to present my reflections as an artist and musician 
performing with an artificial intelligence musical instrument. The question of the un- 
usual musical expectations and further musical demands was also discussed. My main 
intention has been to introduce the unfamiliar musical expectations that appear through 
the performance of the composition Uncertainty Etude #2. In this composition the inte- 
gration of an artificial intelligence method with the creative practice of a musician con- 
tributes to the autonomous structure of the piece. The performance of such a composition 
involves further challenges for the musical expectations and musical demands— both for 
the human musician and the audience. I intended to discuss in what particular way musi- 
cal expectations and musical demands become present through an autonomous behaviour 
that was built to enable the appearance of alternative sound-producing expressions, which 
are layered in the changing audio features of the generated audio samples. I hope this 
article will contribute to an ongoing discourse about new creative technologies, and es- 
pecially to the debate around the use of AI technology in music practice and a new way 
of thinking about composing and performing with musical instruments. 
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PART II 


EMBODIED OBJECT 


#5 
TIMESCALES FOR SOUND-MOTION 
OBJECTS 


ROLF INGE GOD@Y 


5.1 Introduction 


There can be no doubt that Pierre Schaeffer was one of the most influential figures in 
20th century European music, both as a composer of electroacoustic music and as the 
developer of a new and extensive theory of music. We are much in debt to his artistic and 
theoretical achievements, and in spite of more recent advances in music technology, we 
can still find his contributions highly relevant for music making. In particular, Schaeffer’s 
idea of sound objects as the basis for musical creation, perception, and music theory 
(Schaeffer 2017; Godgy 2021), was a remarkable change of paradigm in Western musical 
thought, and this sound object paradigm is still highly relevant for music theory, as will 
be highlighted in this chapter. 

In brief, the sound object can be defined as a holistic mental image of a fragment 
of sound, typically in the approximately 0.5 to 5 seconds duration range, and the sound 
may be of any origin, vocal or instrumental, electronic or environmental, and from any 
musical culture. The focus on sound objects as the basis for both creation and theory, 
this chapter will argue, has the following advantages: 


a) Universal in scope, applicable to very different kinds of music 

b) Holistic, making the object-timescale features accessible for scrutiny 

c) Salient musical features (style, motion, affect) can be found at the object timescale 
d) Human motor control seems to function optimally at the object timescale 


The main point of this chapter is that the object timescale has a privileged role in 
music, both in terms of intrinsic sound features and the associated body motion fea- 
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tures. The main source for this view of the object timescale is Schaeffer’s monumental 
Traité des objets musicaux (Schaeffer, 1966), now also available in English and referred 
to as (Schaeffer, 2017). Other sources include Michel Chion’s Guide des objets sonores 
(Chion 1983), an excellent overview (endorsed by Schaeffer himself) of the most impor- 
tant topics of Schaeffer’s Traité (also available in English as Chion, 2009), and Schaeffer 
and colleagues’ Solfege des objets sonores (Schaeffer et al., 1998), a collection of three 
CDs with sound examples and music theory topics narrated by Schaeffer. This work is 
very useful because of the sound examples, and can make Schaeffer’s ideas accessible for 
a broader audience. For practical reasons, references to these sound files in this chapter 
will be given as follows: ‘Solfège, CD#, track#’. 


5.2 Schaeffer’s object focus 


The idea of sound objects as the basis for a music theory emerged from practical work 
with concrete music (sometimes referred to with the French label, musique concrète), 
when composers, before the advent of the tape recorder, used looped sound fragment 
recordings, i.e. closed groove (sillon fermé in French) on phonograph discs, to combine 
different sounds in compositions. As the composers listened to innumerable repetitions 
of such sound fragments, they discovered that their attention shifted from the immediate 
and everyday signification of the sound to the more qualitative features of the sound, 
i.e. to the features that came to be the content of the so-called typology and morphology 
of sound objects, the elaborate scheme for feature classification in Schaeffer's theory. 
Later, Schaeffer and his collaborators came to realize that their modus operandi during 
the early years of the musique concréte had involved a phenomenological shift in focus, 
known as epoché in the writings of Husserl (Husserl, 1982). In retrospect, Schaeffer 
called this ‘doing phenomenology without realizing it’ (Schaeffer, 2017, p. 206). We can 
now see that Schaeffer's music theory has several other affinities with phenomenological 
philosophy, such as the procedure by sketches and the object-centred view of perception 
(see Schaeffer, 2017, p. 210). 

The shift of focus from everyday significations (e.g. the squeaking door signalling that 
someone is coming) to the more qualitative features (e.g. the overall dynamic envelope 
and the upward glissando of the squeak sound), was called reduced listening, and it 
should be emphasized that this was a method for exploring sound features. This reduced 
listening was related to the idea of acousmatic music, ‘acousmatic’ here denoting music 
emanating from loudspeakers with no other visible sound source. Furthermore, a close 
reading of Schaeffer will show that any sound object is ontologically composite, i.e. it 
will usually have several different features and significations in parallel, but the overall 
dynamic and spectral features form the basis for the typology of sound objects, enabling 
an initial and coarse classification of sound objects based on their dynamic and spectral 
shapes. 

The research method of Schaeffer was that of starting out with seemingly naive ques- 
tions of what we are hearing, with a kind of Socratic approach of top-down scrutiny of 
what is in our minds. In the words of Schaeffer, this could be summarized as 'exploring 
the listening consciousness' (Schaeffer, 2017, p. 109), and with questions of the overall 
features of the sound objects like: What is the dynamic shape of the sound object? What 
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is the mass (subjective sense of spectral shape and/or pitch) of the sound object? Is the 
sound object stable or does it fluctuate? Is it sustained or more impulsive? 


A crucial point here is that the sound object is not a static entity, but first of all a 
mental image of a fragment of unfolding sound. Much effort in the writings of Schaeffer 
is devoted to what the sound object is not, and emphasizing that the sound object is a 
mental image resulting from attentive perception across multiple listening experiences, 
as well as being ontologically composite with a multitude of features in parallel. 

Another vital point here is that a sound object may have a non-linear relationship with 
its acoustic basis, i.e. there may be a relationship of so-called anamorphosis, or warping, 
between the acoustic signal and the mental image. This often non-linear relationship be- 
tween the acoustic features and the subjective percept is due to some perceptual-cognitive 
factors, primarily the following: 


* The mutual influence of the parts of a sound object unfolding in time, i.e. the attack 
part colouring the sustain part and vice versa, or to what extent the sound object's 
identity is preserved or not across different cuts in its unfolding. 


* Differences across the spectrum of what we perceive as a coherent instrument, e.g. 
if we shift the spectrum of a deep piano tone up a couple of octaves, it sounds more 
like a harpsichord than a piano. 


The point with anamorphosis is that there may not be a one-to-one relationship be- 
tween the acoustic features and our subjective perceptions. In Schaeffer's method, this 
means taking our perceptions as primordial and not regarding perception as flawed, but 
instead exploring the correlations between acoustics and perception, correlations that 
also take this anamorphosis into account. 

What is crucial here is the internal coherence of the sound object in the sense of 
temporal bi-directionality, i.e. that present is tainted by past and past is tainted by present 
(as well as by future expectations), as was concretely documented by Schaeffer with the 
so-called cut bell experience, which showed how removing the attack segment could 
totally alter the sound of a bell. This past-present-future tainting is yet another reason 
why the closed groove is such a powerful tool for research, as it documents the workings 
of context at the sound object timescale. From this primacy of the subjective perception, 
the next step was to study the dynamic shapes in the so-called typology, extended also 
to pitch and spectrum-related shapes, as well as later to various internal features of the 
sound object in the so-called morphology, and the combination of these in the typology 
and morphology summary diagram (Schaeffer, 2017, pp. 464—467). 


5.3 General object cognition 


The focus on objects in perception and cognition is also found in other domains of 
thought (see e.g. Shinn-Cunningham, 2008; Starrfelt, Petersen, & Vangkilde, 2013; De 
Freitas, Liverence, & Scholl, 2014 for some interesting cases), and it could be useful to 
have a quick overview of some generic ideas on object-centered perception and cognition 
to see how they may contribute to the idea of sound objects. 
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Two strands of thought in the late 19th and early 20th century stand out: Gestalt 
theory, and the previously mentioned phenomenological philosophy of Husserl. From 
the pioneers of Gestalt theory such as von Ehrenfels, Stumpf, Wertheimer, Koffka, and 
Kohler (Gust to mention the most prominent ones) up to Bregman in the late 20th century 
(Bregman, 1990), the idea of holistic perception and cognition has been prominent, and 
principles such as belonging and exclusive allocation have contributed much to under- 
standing what we call object cognition. Another relevant case of Gestalt theory is that of 
motor control (Klapp, Nelson, & Jagacinski, 1998; Klapp & Jagacinski 2011), suggest- 
ing that body motion can be understood as consisting of pre-planned chunks similar to 
gestalts in perception. It could also be suggested that motion chunks contribute to sound 
object formation, as is the point of so-called motormimetic cognition (Godøy, 2003) and 
the idea of extending Schaeffer’s typology categories to body motion (Godøy, 2006). 

In parallel with early Gestalt theory, the phenomenological philosophy of Husserl 
provided important contributions to the epistemological reasoning about objects in our 
experiences. According to Husserl, we need to step out of the continuous stream of 
sensations in order to constitute meaning, and this happens by means of a series of so- 
called now-points, i.e. points in time where we interrupt streams and lump together past, 
present, and expected future sensations into somehow meaningful chunks (Husserl 1991; 
Godøy 2009). 

In an experimental vein, we have seen research (since the pioneering work of Miller 
(1956) on chunking) on the workings and effects of different kinds of chunking in human 
behaviour (Gobet et al., 2016), and more neurocognitive views on chunking and sensa- 
tions of presence can be found e.g. in Póppel (1997), in Varela (1999), and in Wittmann 
& Póppel (1999). Additionally, we have seen work on perception and cognition of audi- 
tory objects by holistic integration of sensory data (Bizley & Cohen, 2013); lastly, there 
is an extensive theory of objects and shapes as fundamental elements in human reason- 
ing to be found in morphodynamical theory (Thom, 1983; Petitot, 1985; Petitot, 1990; 
Godøy, 1997). 


5.4 Sound object features 


What constitutes a sound-motion object is first of all a sense of energy shape, of starting 
and ending within a timespan of approximately 0.5 to 5 seconds (sometimes longer, but 
that is more exceptional). This overall energy shape, or envelope, is one of the main 
features of Schaeffer's typology, the typology being a first and coarse sorting of sound- 
motion objects. The main categories of these energy envelopes, called facture in French, 
a term designating the way they are made, are as follows: 


= Sustained, a continuous, quasi-stationary sound 
= Impulsive, a brief sound, e.g. percussive or plucked 
= Iterative, a rapid series of onsets, like in a tremolo 


But also a first and coarse sorting of sound content is included in the typology with 
the three categories of so-called mass, denoting the frequency domain (and not just per- 
ceivable pitch) as follows: 
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* Tonic, more or less clearly perceivable pitch, relatively stable 
= Complex, a composite sound, inharmonic, or noise dominated, but stable 


* Variable, changing in perceived pitch or spectral placement 


These mass and facture types are combined in a 3 x 3 matrix, and can be applied to 
most types of sounds within the typical duration range for sound objects, and several 
examples of these categories can be heard in the Solfège, CD3, tracks 19-42. 


There may be so-called phase transitions between these typological categories, e.g. 
an impulsive sound may turn into a sustained sound if extended beyond some duration 
threshold, or an iterative sound may turn into a series of impulsive sounds if the sound 
is slowed down, etc.; in short, there are categorical thresholds here due to duration and 
density of events. And importantly, these musically meaningful categorical thresholds 
are all related to sensations of body motion, to both motor control and bodily effort. 
The overall energy envelopes of sound objects are usually the most perceptually salient, 
capable of triggering sound-accompanying body motion in listeners, such as in various 
cases of so-called entrainment (Clayton et al., 2013). 

There are in addition a number of internal features of the sound objects in what 
is called the morphology, organized in a system of main feature dimensions, each in 
turn with sub-feature dimensions, and sometimes also with sub-sub-feature dimensions. 
These dimensions concern the frequency domain features and their various sub-features, 
and some dynamic features and their sub-features. The most important are those con- 
cerned with textures: 


= Grain includes various instances of very fast fluctuations in the sound, such as in a 
trill, yet not so fast as to enter into the audio region (i.e. not above = 20 hz). 


= Gait denotes slower kinds of fluctuation, e.g. such as those found in dance, walking, 
and other body motion patterns. 


There are also other morphology dimensions such as mass (overall spectral content), 
dynamics (overall loudness), harmonic timbre (spectral distribution), melodic profile 
(pitch-related shapes), and profile of mass (spectral shapes); however, the typology con- 
tains the most prominent features for sound-motion objects, because it includes shapes 
at the sound-motion object timescale. 

Large-scale forms may also be conceived of as objects, as has often been the case in 
Western music theory, but Schaeffer was clear that his focus was on the materials of the 
sound objects and not on large-scale works (Schaeffer, 2017, p. 17). Schaeffer introduced 
the concept of the suitable object: neither too short nor too long, and in practice (as 
evident in the examples of the Solfège) typically in the 0.5 to 5 seconds duration range, 
as well as some other criteria of information density, i.e. neither too complex, nor too 
simple, to keep the attention of the listener. 

With too-long fragments, it would not be possible to focus on salient features, because 
they would change. With too-short fragments, perceptually salient features would not 
exist (would not have time to become manifest). In exceptional cases, sound objects in 
the Solfége last for up to 30 seconds. But the main duration criterion is that of content, 
ie. that the sound object manifests a clearly perceivable salient shape. This means that 
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the typological categories mentioned above become the main criteria of timescales, as 
can be heard in Solfège, CD3, track 19-22, as well as examples from various sources, 
instrumental and electronic, in the Solfège, CD3, track 22-42. To make this point about 
suitable objects, we are in Solfège, CD3, track 42-59 presented with other examples of 
sound objects deemed not to comply with suitable objects criteria by being either too 
long, too short, too dense, having too much redundancy, being too unpredictable, or too 
chaotic. 


5.5 Continuity vs. discontinuity 


What the preceding object principles boil down to is the relationship between continu- 
ity and discontinuity in musical experience. The question is: To what extent does our 
organism work by continuous or discontinuous processes and decisions? For more gen- 
eral points of view on this, see e.g. Miller (1982), Sergent and Dehaene (2004), Spivey 
(2008), and Reybrouk (2021); but in motor control, continuity vs. discontinuity has been 
debated for more than a century, and has engendered various models of how we plan, 
trigger, and control body motion in different contexts (Elliott, Helsen, & Chua, 2001). 

The conundrum of continuity and discontinuity in motor control can in a sense be 
bypassed by seeing how triggering and control may be discontinuous, while the results 
may yet be continuous, i.e. the motion trigger may happen at discrete points in time, 
but the resultant motion may be extended over longer stretches of time. Discontinuity 
in motor control is often believed to be based on constraints, i.e. that our organism 
is too slow to enable continuous control, and that continuous motion is an emergent 
feature of our organism's implementation of discrete control impulses, as suggested by 
the following: 


= Klapp and Jagacinski (2011), with discontinuous action gestalts (as mentioned above) 
resulting in seemingly continuous body motion. 


* Grafton and Hamilton (2007), with discontinuous control through command hierar- 
chies resulting in continuous motion and also emergent coarticulation, i.e. a contex- 
tual smearing of otherwise separate motion units into more continuous motion. 


= Rosenbaum (2017), suggesting that motion is controlled by so-called goal postures 
with continuous body motion emerging from transitions between distinct postures. 


= Sosnik et al. (2004), demonstrating how initially discontinuous, point-by-point mo- 
tion may turn into more continuous smooth motion by coarticulation. 


= Godøy (2014), arguing how coarticulation is at work in sound-producing body mo- 
tion, resulting in a contextual smearing of both the body motion and the resultant 
sound. 


An interesting development in motor control here is the theory of so-called intermit- 
tent control. This theory suggests that human body motion may be controlled discon- 
tinuously, i.e. in a point-by-point manner called serial ballistic control (Loram et al., 
2011). The reason for this serial ballistic control scheme is again that our organism 
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seems to be too slow to enable totally continuous control, and needs to work by antici- 
patory control, i.e. in a point-by-point manner by so-called open loop control. However, 
the perception of the output motion may be more continuous, hence the idea of ‘observe 
continuously, act intermittently’ (Loram et al., 2011, p. 317). Intermittent control is an 
ongoing research topic, but we may hypothesize that it could be a plausible way to rec- 
oncile continuity and discontinuity in sound-producing body motion as well, and hence 
also in our mental images of sound-motion objects. 


5.6 Concluding remarks 


The mentioned categories and criteria of a combined sound-motion object theory could 
make us aware of, and give names to, various perceptually salient features of the music. 
This could provide us with both analytic and generative tools, particularly useful for 
enhancing our grasp of the different sound-motion object types derived from Schaeffer’s 
work (see Godgy, 2018 for details), such as: 


* Composed objects, combining different components by juxtaposition, additively en- 
hancing sound-motion objects. 


= Composite objects, denoting components in a sequence, fused by coarticulation into 
more complex extended sound-motion objects. 


Beyond the suitable objects mentioned above, there are also various other objects that 
can be named, and there are very many possibilities of object combinations and concate- 
nations into more extended collage compositions. All this can have useful applications 
in musical analysis as well as in creative tasks, such as: 


» Sound design and orchestration by systematic combinations of typological features. 
* Composition and Improvisation as scripts with concatenated typological shapes. 


Throughout these processes, thinking sound-motion objects and various typologi- 
cal and morphological features, could be a systematic approach to handling otherwise 
ephemeral material. 

Hopefully, sound-motion objects, combining the remarkable insights of Schaeffer and 
his collaborators from more than half a century ago with current research on music- 
related body motion, could be a vehicle for further exploration of the holistic nature of 
musical experience. This could be expressed as a theory of musical quanta (Godøy, 
2013), denoting a holistic object-centred approach combining the overall sound and mo- 
tion features in a way that also reconciles continuity and discontinuity in musical experi- 
ence. 

Notably, such an object-centred approach would not exclude more macro-level ap- 
proaches as can be seen, for instance, in projects of music information retrieval. Explor- 
ing features of larger collections of sound-motion objects or of more extended, macro- 
level works of music, e.g. spectral centroid, spectral flux, harmonicity, etc. using avail- 
able software e.g. the MIRtoolbox (Lartillot & Toiviainen, 2007), could be combined 
with the more local sound-motion object feature studies. This could also provide us with 
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quantitative information about various typological and morphological features, providing 
the acoustic correlations of these subjective features as was indeed the stated long-term 
aim of Schaeffer’s music theory. 
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#6 
MULTI-FORM VISUALIZATION: A METHOD 
TO COMPOSE ACOUSMATIC MUSIC 


VIRGINIE VIEL 


6.1 Introduction 


In acousmatic music, the sonic realm, traditionally qualified as immaterial and ungras- 
pable, is manipulable and concrete, thanks to Schaeffer’s (1966) examination of musical 
listening and the factors of psychological conditioning that influence it. The composer’s 
practice of sound recording, and her ability to intervene in the structure of sound in ways 
that transcend the actual physical process of sound production and its sound source, chal- 
lenge the instinctive and natural tendency to draw connections between sonic and visual 
stimuli. When the actual physical sound sources are absent and sound is apprehended 
solely through loudspeakers, the listening experience becomes acousmatic. In this ex- 
posé, an examination of the perceptual process allows us to examine the perception of 
qualia in music. I then demonstrate how conceiving the perceiver’s mind as embodied 
is key in the development of my compositional practice, which is based on the idea of a 
multiform visualization. 


6.1.1 The perceptual process 


The process of perception is complex and has relevance to philosophy, cognitive sciences, 
anthropology and geography. The primary sense of proprioception which, through con- 
tinuous sensations triggered by the muscular activation of the body, helps us to be aware 
of our physical existence as organized entities. Therefore, perception is a first person 
experience (Lamedica, 2014) and the perceiving body is the origin of an ego-centred sys- 
tem of coordinates that evolves in an environment (De Monticelli, 2013). Delivery of 
information from our sensory apparatus, such as kinaesthetic sensations, makes us con- 


https:/doi.org/10.34727/2022/isbn.978-3-85448-047-1 6. This work is licensed under the 
Creative Commons Attribution-ShareAlike License 4.0 International (CC BY-SA 4.0). 
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scious of our position within the environment such that we can actively explore the world 
around us. 

According to Fugali (2013), the sensation of being physically anchored in the world 
relies essentially on touch, which is exteroceptive! and interoceptive?. That is to say, 
touch is both outwardly and inwardly oriented, providing information about the world's 
material features and an awareness of our bodily posture, balance and inner condition. 
The perspectival way we gain access to the world—determined by the dynamic inter- 
action between our sense organs and defined as the mobile parts of the sensory sys- 
tem— shapes our perceptual experience (Gibson, 1983) . 

The lived environment provides us with different forms of stimuli that we generally 
define in regard to the different sensory modalities that characterize our body: visual, 
aural, tactile, olfactory or gustative. Our sensory modalities represent the different chan- 
nels through which information emitted from the physical environment is received by the 
perceiver. Once information has been processed, the perceiver, in turn, sends a bodily 
response to the environment through one or several sensory modalities. Therefore, the 
perceptual process can be defined as interactive. Gibson (1987) considers perception as a 
direct, mutual and continuous relationship between an organism and its environment. The 
complexity of our constant interaction with the world around us invites the interweaving 
of consciously and subconsciously collected and processed information. Thus perceivers 
constantly adjust their own set of perceptual skills to the features of their environment. 


6.1.2. Sensory engagement 


Cognitive sciences see the nature and functioning of our mind as mainly determined by 
our bodily constitution and the way we engage our sense organs in an active exploration 
of our environment. O'Regan and Noé (2001) conceptualize the sensorimotor aspect of 
our perceptual process to examine the nature of our sensory engagement. They observe 
perceptual experience as a mutual co-determining relationship between body and envi- 
ronment and explore the idea of phenomenal feel (Torrance, 2005) through two features: 
corporeality? and alerting capacity^. These two properties help to elaborate sensory 
consciousness. They are based on the skills involved in perception—namely the articu- 
lation of specific patterns of sensorimotor interdependence between perceivers’ sensing- 


! A sensory modality that is outward-oriented. It turns to an object in the external world or to material features. 
2A sensory modality that is inward-oriented. It has a self-reflective structure, which allows the perceiver to be 
bodily self-aware. 

3A feature used by O' Regan and Noé (2001, pp. 939-1031) to define phenomenologically perceptual experi- 
ence as a mutual co-determining relationship between body and environment. It is a characteristic whereby 
bodily movements affect how the environment offers information to our sensory apparatus. For instance when 
I observe a forest from far away or from inside, it is clear that my bodily position influences the way I relate 
to the forest. From a general perspective, a corporeal quality can define anything that has a body or material 
existence. 

^A second feature used by O' Regan and Noé (2001, pp. 939-1031) to define perceptual experience phenomeno- 
logically as a mutual co-determining relationship between body and environment. It considers any perceived 
change in the environment, like the arrival of the rain, as an invitation to take a sensorimotor action and adjust 
my bodily attitude, for instance by covering my head with my umbrella to continue observing the forest. In this 
situation, I adapt my bodily position to the environmental change to remain focused on the forest as the object 
of my attention. 
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moving body and their environment. But is this theory applicable to acousmatic music 
listening? 

In fact, when I listen to a piece of music, my bodily engagement towards the music in 
an acousmatic context is fundamentally different to the perception of my multi-sensory 
environment. The mono-sensory quality of my acousmatic experience—for example, I 
am listening to music, sitting on a chair in the darkness of the concert hall or in the 
studio—does not require the use of my sensorimotor skills as much as when I perceive 
the environment. Instead, it invites me to engage in a mental process and to adapt the 
focus of my attention through my cognitive skills. In these terms, a helpful way to widen 
the perspective offered by O’ Regan and Noé is to examine the sensory dimension of the 
perceptual experience and to look at the concept of embodiment to articulate the impact 
of bodily experience in musical perception. In the context of an acousmatic concert, the 
corporeality and alerting capacities are not literally applicable to bodily engagement, yet 
they may be useful when transferred into the cerebral domain, using mental projection. 


6.1.3 The embodied experience 


In phenomenological terms, embodied experience is achievable in a unique and struc- 
tured moment of presence. It intrinsically connects the perceived object to the perceiver 
in an interactive and constructive mediation. I am bodily aware of the multi-sensory 
structure of my living environment thanks to the inextricable intertwining of my senses 
rather than by the independent action of each sense organ. The combination of stim- 
uli, either modal (visual, tactile, smell, aural, olfactory) or bodily perceived (proprio- 
ceptive, kinaesthetic, vestibular) captured during my sensorimotor exploration and then 
processed, provides essential information to phenomenally ground the perceiver’s body 
in a transmodal> experience and understand the world in which I live. For example, when 
I look at a flying bird, an active sensorimotor cooperation between sense organs occurs. 
The capture of supra-modal information (such as proprioceptive, kinaesthetic, extero- 
ceptive ones) and visual information positions my body as a stationary and grounded 
self while I perceive the bird as a non-self body, moving in the air. Each modality ex- 
plores the environment in its own style, picking up and pairing multi-modal information. 
Reflection on how this multi-modal information can be held together in the image of a 
world apparently coherent, solid and meaningful like our living environment becomes an 
important way in which we explore the nature of our senses themselves and the way they 
contribute to our understanding of the world and our place in it. 


The notion of embodied mind is conceived as relational, distributed over body, brain 
and environment (Scarinzi, 2014), without being attached to any physical structure. From 
this perspective, the sensorimotor contingency theory seems to apprehend only partially 
the whole experience of life because it focuses just on objective instrumental actions 
without accounting for the non-physical aspect of life experience, which is the organism’s 
subjectivity. Our subjectivity, which gives its unique colour to each of our experiences, is 
according to Fuchs necessarily embodied (ibid., p. 74), that is to say, conditioned by how 


5Perceptions in which one sensory modality triggers those of another. For example, sound and vision can 
provide me with spatial, tactile or kinaesthetic information, whose intermeshing imbues a particular scene, or 
a landscape, with multi-modal qualities. 
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our body is shaped and how connections between senses are physiologically constructed. 
So in relation to the idea that our perceptual experience relies on inner representations 
of the external world, as enactivism positions, our body determines our subjectivity—the 
way we singularly interact with the world around us through perception, thought, feel- 
ings, desires and imagination. This defines us as autonomous and self-determined selves 
capable of generating meaning out of our experiential interactions with the world, re- 
vealing themselves as an entanglement of objective and subjective experiences. In this 
regard, understanding the notion of qualia® (Godøy, 2006) is key to understanding my 
compositional method. 


6.1.4 Musical perception: vectors of qualia 


The subjective sensations that accompany sensory experience are referred to as qualia. 
Our perceptual activity, thoughts and feelings as ensured by the constant dialogue be- 
tween sensory information, practical knowledge and our subjectivity—are all contribu- 
tary to the emergence of meaningful experience. 

Like the world around us, a piece of music is essentially an object in constant motion, 
composed of intermeshed, sequential, overlapped gestural formations (Godøy, 2006). 
Our knowledge of the sounding world is rooted in motion, fundamentally based on our 
sensorimotor skills and bodily experience, which can be interpreted as gesture. In lis- 
tening to acousmatic music devoid of anecdotal content we can still experience a certain 
degree of connection to and familiarity with the shaping influences and structures of the 
environment. 

According to Smalley (1997) this could be due to the ability of sounds to commu- 
nicate proprioceptive and motivating sensations that refer to our personal experience of 
sounding gestures’ through the perception of their spectromorphology?. 


Perceiving any qualia in music relies fundamentally on our body-centred and ecologi- 
cal perceptual behaviour, led by our instinctive tendency to seek coherence and meaning 
though connection between the unknown and our experiential knowledge of the world. 
Thanks to the electroacoustic tools available to composers to record, create and alter 
sounds, pieces of electroacoustic music give listeners the possibility to phenomenolog- 
ically perceive infinite numbers of typo-morphologies? and spectromorphologies. The 
relationship between bodily experience in the living environment and music perception 
inherently colour the way listeners may experience various qualia in musical listening, 
invoking our experiential knowledge of the physical properties of our living environment. 

For example, the perception of qualia of motion could give the feeling one is virtu- 
ally witnessing the making of a bodily gesture of a cellist to create a sound, or that one 
is observing a dancer moving in space. Following Smalley's thinking the propriocep- 


Qualia results from our capacity to feel and subjectively experience the materiality of the world thanks to our 
embodied mind and the dynamic performance of our senses. They are subjective sensations that accompany 
sensory experiences. They are personal and phenomenal (Huron, 2006). 

?Smalley (1997) defines gesture as an energy-motion trajectory which excites the sounding body, creating 
spectromorphological life. 

5Smalley (1997) defines spectromorphology as the interactive relationships between sound spectra (-spectro) 
and the way they change and are shaped through time (-morphology). 

?The typo-morphology has been defined by Schaeffer (1997). It analyses and categorises each sound based ist 
type and a detailled description of its morphology. 
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tive sensation of gesture aurally perceived refers specifically to a sonic gesture related to 
physical sound sources like, for example, the opening of a door or footsteps. Sometimes, 
the motion perceived is not explicitly familiar and may mislead the listener’s understand- 
ing of the composer’s intentions and also lead her to forge imaginative connections. This 
is particularly the case when sound materials are abstract and their spectromorphology 
not “firmly rooted in known sounding gesture’ (Smalley, 1996). The qualia of physical 
presence are communicated through qualia of distance that can be associated with visual 
perception of the depth of field—both phenomenologically. 

The quale of substance, perceived phenomenologically, is fundamentally based on 
spectromorphology, and features the proprioceptive sensations of density, mass and tex- 
ture that relate to very intimate sensations of touch, invoking our experiential knowledge 
of the physical properties of our living environment, such as liquidity, hardness, smooth- 
ness, or ruggedness. The notion of substance also echoes the Schaefferian notion of 
facture. However, this category of quale, less abstract than the other quale of physi- 
cal presence, seems to be more connected to the materiality of our living environment 
and draws a substantial link to the recognition of anecdotal features, or physical sound 
sources, triggering mental images that connect instantly to our multi-sensory experience 
of the world. 


6.2 In my practice: the capacity to imagine 


The experience of listening to acousmatic music is a multi-layered process. It starts in 
the aural domain with the perception of sounds, which I generally consider either as 
abstract or as variably representative or evocative of my experiential knowledge of the 
sounding world—that is to say, through recognition of particular physical sound sources 
(like the sound of running water or a voice) or bodily sensations (for example, qualia of 
motion or substance). Listeners rarely perceive acousmatic music as exclusively abstract, 
anecdotal or figurative. Their experience is a continuous attunement to their feelings 
about what they perceive, supported by a constant dialogue between phenomenological 
features presented by a piece and their embodied mind. As a listener and a composer, I am 
naturally drawn to acousmatic pieces that feature abstract rather than explicitly anecdotal 
sound materials, as they engage listeners in a ‘feel-relationship’ with music based on 
perception of qualia. This has the potential effect for listeners of immediately engaging 
their being in perceiving transmodal sensations, rather than calling on a response based 
primarily on the cognitive process of sound source recognition. When I compose music, I 
see each project as an opportunity to create a new sensory experience, inspired by mental 
images based on perceptual experiences—teal or imaginary—trather than inspired by the 
sound materials I have in my sound library or others I may like to record. 

From a general perspective, my sources of inspiration are multi-modal—I conceive 
vision, touch and hearing as channels or containers capable of transmitting similar mes- 
sages in different ways. That is why my source of inspiration can be as broad as the 
qualia of vast space, the qualia of the motion of the sea, the vision of darkness, the tactile 


0] the Schaefferian typology facture is a criterion to describe the way that energy is communicated and 
evolves through time. It determines whether the energy of a sound is maintained continuously, iteratively or 
weakly. 
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sensation of a fizzy texture, the qualia of violent gesture, and so on. To tackle and im- 
plement these mental images into music, my method consists in analysing them through 
tables of key words, reducing them through sketches to explore how I could feel them 
through my bodily gestures and associating them with video clips whose observation be- 
comes a way to absorb and remotely feel them. All these methods help me to make them 
‘mine’, and to associate them with a spectromorphological design, an element of sound 
material, a process, a sequence or even a way to structure the form of a piece. The shap- 
ing of sound materials, like the visual dimension of my working process, is conducted 
using this feel-relationship. The sounds I choose and shape also profoundly influence the 
way I compose sound sequences by triggering new mental images. They emerge in turn 
from the perception of the phenomenological features of the processed sound materials 
and the sequences in progress. Therefore my compositional process can be perceived as 
a constant and recursive dialogue between mental images, qualia and phenomenological 
features, using the multi-sensory dimension of my bodily experience as an inspirational 
tool to compose music that is evocative of this multi-sensory dimension of human per- 
ception. I call this method the multi-form visualization. 

In this very personal way of mapping and projecting mental imagery onto a practical 
compositional approach, the coupling of mind-body then becomes central to the compo- 
sitional and listening experience in acousmatic music. 

The ability to imagine is essential when the gestures we detect via spectromorpho- 
logical characteristics are ‘firmly rooted in known sounding gestures’ (Smalley, 1996) 
or, on the contrary, when the spectromorphology has been so much processed that its 
initial gestural impetus becomes dissolved, losing all intimate connection it could have 
with our personal experience of life, we may feel completely detached from a work, 
‘as if observing invisible phenomena displayed on a screen or in space’ (ibid.). Yet a 
feature of acousmatic music is that composers choose to deliberately shape a listening 
experience with no equivalent in the real world, leading listeners to build bridges be- 
tween phenomenological features of sounds, associated bodily sensations and the mental 
images they experientially or imaginatively connect to sounds in order to find meaning. 
Moreover, when no realistic connection can be drawn between a piece and our physical 
environment, trying to picture a meaningful sound image of it cannot only be based on 
our bodily experience of the world but must also be supported by our ability to imagine. 
Indeed, a particular texture, the impression of a gesture, or a particular pitch can invite 
us to experience a conscious dream (une réverie consciente (Bachelard, 1971)) that pro- 
gressively influences us and leads us in a flow of thoughts, images, and sensations in our 
mind. Between daydream and the sensory reality of a piece, this type of reverie comes to 
our mind as an alternative space, reassuring and open to the world but also intimate: it is 
a space where the aural sensations communicate sensations through various qualia. 


6.3 The use of multi-form visualization for An Ocean on the Moon 


6.3.1 Building the project 


The use and design of visuals in my practice of composition is central. I primarily use 
sketches—but also images, photos or videos can work together with tables of keywords to 
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help me in visualizing which sound sources and sonic qualities I should employ in a piece 
and how I could shape the spectromorphological evolution of a composition over time. 
For my piece An Ocean on the Moon'' (2017), I also employed multiform visualization 
to stimulate the production of raw sound materials by instrumentalists, giving them the 
freedom to improvise based on multimedia resources I provided to guide them in the 
expression of particular qualia. 

First of all, An Ocean on the Moon is a multichannel piece, composed for a 16-channel 
sound system that recreates the shape of a dome in order to offer an immersive listening 
experience. The sea is the central theme of this project. When I started it, my perspective 
on the sea was very much influenced by all the mental images I had in mind, like the joy 
of being rocked by the waves, the fear of drowning, the contemplation of its beauty or 
the experience of living at the seaside. At the start of the project I recorded various field 
recordings, sketches, videos and photos in order to find an angle to attack my piece. I 
spent time wandering in the city of Corfu and on the seaside at different times of the day 
to be in touch with the diverse atmospheres, colours and moods of my living environ- 
ment at this time. As I accumulated pictures and video clips and repeatedly reviewed all 
of them, I started to connect each of them to particular keywords, representative of their 
sensory features and qualia. This led me to build the following table (Table 6.1) gather- 
ing these features (some are repetitive), which I organized into two different categories, 
corresponding to different energetic temperaments expressed by the sea, namely qualia 
of calm (A) and of violence (B). 

However, rare were the moments when I could perceive a continuous violent or calm 
energy from the beginning until the end of a recording. In fact, the sea movements 
were very much similar to a breathing pattern, a sort of oscillation between calm and 
violence. Inspired by my observations, I took a pen, a piece of paper and let myself 
dive into the realm of my imagination where I reduced the sea to a phenomenon of 
pure motion. Drawing this line (Figures 6.1?) was for me a revealing step because it 
made me experience proprioceptive sensations that I could relate to a sea wave, like 
continuity of motion, fluidity, freedom, softness, calm, or impetus of expansion. This 
led me to consider this waveform as the form of the piece, and furthermore invited me to 
contemplate how I could consider the temporal and spatial dimension of music in relation 
to this waveform and how I could integrate the opposition between qualia of calm (A) 
and of violence (B) (Figure 6.3). 


11 An Ocean on the Moon is an acousmatic piece that explores a maritime theme through the evolution of a soft 
yet shiny, joyful and destructive instrumental netting texture. A particular emphasis is placed on the diversity 
of sensations that may overwhelm a viewer in front of the sea, between memories of joyful moments, lonely 
wandering and sudden fear. This piece is an evocation of visual and kinetic experiences of the sea. Through 
this project, I studied the rich expressivity of instrumental gestures, sometimes methodically, sometimes im- 
pulsively. I also explored composerly gestures, which at each stage of the creative process works from the 
fluidity of drawn shapes: curved, voluptuous, and sometimes extreme. This project resulted from collaboration 
with several instrumentalists who performed musical sequences based on visual images and video clips. It was 
partly composed at the Ionian University, located in Corfu, Greece in the context of an Erasmus+ Exchange and 
it was completed in the Music Technology and Innovation Research Centre, De Montfort University Leicester, 
United Kingdom. 

!2The pattern of Figure 1 represents an oscillatory movement between qualia A (black curved line) and qualia 
B (red straight lines) evolving over time. The x axis refers to time, the y axis does not refer to any sonic 
parameters. While the curves evoke a cyclic, evolutionary, round and soft motion, the straight lines aim to 
illustrate percussive, aggressive and repetitive sounds 
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Sensory features Qualia 
Drops Shimmering Massive Fragility Violence 
Streams | Fluidity Round Process Danger 
Strong Transparency | Coloured shades | Disorganization | Softness 
Rolling | Protection Curved lines Wildness Cycle 
Slow Rocking Silky textures Craziness Caress 
Wave Reflection Tempestuous Safety Sharp 


Table 6.1: Sensory features and qualia featured in the visual materials collected for 
An Ocean on the Moon 


il | 


Figure 6.1: First attempt to represent the 
breathing cycle of the waves 


Figure 6.2: Curved line, resulting 
from a gesture 


Slowly, the shape, the sound qualities of the piece became clearer in my mind. I 
imagined a blending of instrumental and vocal sounds that would create various textures, 
gestures and motions, methodical and impulsive. In this regard, my collaboration with 
performers was key, as it allowed me to observe and integrate in the project musicians 
whose sensitive responses to the same multimedia resources and mind-maps of keywords 
were different. These mind-maps were essentially used as a trigger, and invited them to 
explore how they could gesturally and sonically embody the mental images, the imag- 
inary or experienced multi-sensory sensations they associated with each mood. Each 
recording session was individual and consisted of seven improvised performances of 2-3 
minutes each. Each resource was related to either qualia of calm or qualia of violence. I 
divided the qualia of calm into three different moods and the qualia of violence into four. 


6.3.2 The structure of the piece 


The recording sessions provided me with a significant quantity of sound materials. I 
created different working spaces in the digital software, called Reaper, that I was using 
to compose the piece. This allowed me to mix and compose sound clips to see which 
energy and sound quality would result from this. Then I started composing different tex- 
tures without applying much sound processing. In this process, I was quickly confronted 
by different types of gestures regardless of the type of mood. They reflected the greater 
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(b) Qualia of violence (B) described as Mood B-1, Mood B-2, Mood B-3, Mood B-4 


Figure 6.3: The mind-maps used during the recording sessions: qualia of calm (sub- 
figure a) and qualia of violence (subfigure b) 
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or weaker fluency of each performer in improvising—some sequences were short, fea- 
turing expressive, assertive gestures with a strong sound presence (drums, harpsichord, 
violin and voice) while other sounds reflected hesitant gestures, a lack of dynamic and 
not much expressivity. I used this project as an opportunity to explore a new aesthetic 
and restrict the amount of sound processing to a minimum in order to explore how dif- 
ferent instruments could interact with one another and create a surprising mix. To this 
purpose, I only used a few effects on my initial sound materials, such as noise remover, 
some filters, pitch, time stretch, and also very little reverberation. Then I started to mix 
these different sequences on Reaper, testing different spatial positioning and automating 
different motion trajectories with Zirkonium!?. I was also very much influenced by the 
performer’s gestures I could hear within the mix. While it was necessary to abandon 
some sequences—because of the way I designed some movements or because their com- 
bination with others did not convince me—I was satisfied with others because I could 
clearly associate them with specific qualia. For example, the combination of different 
timbres, pitches and gestures!^, in some sequences managed to transmit the vision of the 
shimmering and moving surface of the sea, shiny and colourful. 


Figure 6.4: Different textures imagined before starting to record the improvised se- 
quences (horizontal axis refers to time) 


This stage in my compositional process was particularly exciting. The sound se- 
quences I obtained invited me to intuitively rearrange the initial structure (Figure 6.215) 
and refer back to the initial waveform (Figure 6.1) to propose a new structure based on 
this curved line. Following the idea of a flowing gesture, the new structure imaginatively 
describes the deployment of the energy-motion of the waves through time. I illustrated 
this perspective visually through four sketches corresponding to the final four-part struc- 
ture of the piece (Figure 6.5), exploring the dichotomy between qualia A and qualia B. 
I used this presentation as a guide to achieve the composition of this piece. The colours 


'37irkonium is a digital interface employed to spatialize sound. 
14 An example: An Ocean on the Moon (from 0'40 to 1°44) 
‘Figure 2 is a reductive representation of my mental image of the sea, its the temporal motion. 
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used in the figures aim to illustrate the diversity of instrumental timbres I would like to 
use in these different sections. I chose these colours for their visual aestheticism, away 
from any synaesthetic purpose. 


A 


(a) Part One (b) Part Two 
À E 
^ 
(c) Part Three (d) Part Four 


Figure 6.5: Proposal of structure of An Ocean on the Moon 


6.3.3 The spatial composition 


I tackled the spatial composition of the piece as I went along by mixing and composing 
the different textures with a clear idea of which proprioceptive sensations I wanted listen- 
ers to experience throughout the piece. Accordingly, I designed the start of the piece with 
sparse sounds giving the sensations of sparkles of water drops that would come from the 
sky. Falling randomly on us, these sounds would progressively accumulate to create a 
sonic texture that would immerse us in a sound bath, marking the beginning of our jour- 
ney into an imaginary sea world, made of gentle and sharp rocking movements, some 
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voluptuous gestures that evoke sea streams, and even unpredictable spatial trajectories of 
some elements. 

The Table 6.2 synthesizes all the features that inspired the final version of the piece. 
It is a guide that helped me to complete the composition of the piece. It combines the 
structure proposal, a summary of the mental images that emerged while I was listening to 
and mixing the recordings—which part corresponded to which quale (A or B), how the 
notion of knitting, texture and the quality of the sound mix have been thought through, 
and finally, how I conceived the work’s general spatial composition. 


6.4 Conclusion 


Multi-form visualization is a powerful method to assist composers of electroacoustic 
music in their creative process by forming essential reductions of imagined sound into 
something that can be embraced and extended through direct auditory sensation in the 
studio, as a stimulus for production, an aid for the processing of raw sound materials and 
also as a guide to building a piece. It is a method I invite composers of electroacoustic 
music to follow in order to profoundly connect with qualia and also engage in the shaping 
of sound materials, in such a way that their phenomenological complexity leads to an 
intertwining of the realms of memory and the imaginary. 
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of discrete rocking motion 


gestures spread 


across space 


Visual Representation | Figure 6.5 (a) Figure 6.5 (b) | Figure 6.5 (c) Figure 6.5 (d) 
Sensation of . i . 
ES Sensation . Sensation of 
being immersed Upcoming change: PE. . 
f of release . . being in the middle 
in a dense and increase of energy 
of energy of a tempest 
smooth texture 
Mental Images Mix of sparkles | Increasing 
of sounds like sensation : Increase of 
3 Increasing pace . 
drops of water of calm and violence 
and streams peace 
Rocking Progressive 
Hot water ; Danger 
sensation change 
Softness Airy Less roundness Dense texture 
Sharp sound 
Fluidity Colourful Sharper attacks m 
materials 
Succession of 
. . : Sudden emergence 
High density Shiny y heterogeneous 
of diverse sounds 
textural elements 
Qualia A A A-B A 
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1.Start = pointillist organization : : 1. Sensation of 
: immersion : i 
of space = one sound = one $ . immersion 
2. Rocking motion Met 
speaker 2.Combination 
. across space ] 
2. Then, progressive of different 
. 4 (forward — backward i . . 
Spatial accumulation of sounds ] spatial trajectories 
D and left to right A : 
Composition on each speaker: A eae combined with 
i j : : motion) is 
sensation of immersion without : sudden appearance/ 
: emphasized by . 
major gesture across speakers. l disappearance 
. presence of wide 
3. End of part 2 : emergence of sound on 


different speakers 


Table 6.2: Final inspirational guide designed to compose the piece An Ocean on the 


Moon 
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THE OCULARCENTRIC 
OBJECTIFICATION OF MUSICAL 
EMBODIMENT IN COGNITIVE 
CAPITALISM: COVID-19 AS AN 
ALLEGORY ON THE MULTIPLE SENSES 
OF TOUCH 


PAVLOS ANTONIADIS 


7.1 Introduction 


Triggered by the Covid-19 pandemic, this paper attempts a problematization of the no- 
tion of touch in musical performance. The de facto crisis of musical haptics due to 
physical and social distancing is here considered in the context of a wider phenomenon, 
namely the ocularcentric objectification of musical embodiment. This reduction of mu- 
sical embodiment to its visual dimension has been a long, historical process, accelerated 
by the political economy of cognitive capitalism, including Covid-19 as a catalyst of 
pre-existent tendencies. By revealing a crisis of touch, the ongoing sanitary crisis invites 
us to further reflect on the meaning of musical haptics, beyond the visual properties of 
embodied gestures and beyond tactility in the design of tangible user interfaces. In that 
sense, Covid-19 becomes a modern allegory on the multiple senses of touch, similar to 
the allegories of the senses in Flemish Renaissance painting. 


7.2 The ocularcentric objectification of embodiment in contemporary mu- 
sicology 


The role of embodiment in musical performance has been emerging as a central theme 
in the context of musicology’s performative’ and ‘embodied cognitive’ turns.! Tradi- 


lFor a good overview of the performative turn in musicology, please refer to Lalitte (2015); Clarke and Cook 
(2004); and Pace (2017). For the embodied cognitive turn, please refer to Leman (2008). 


https:/doi.org/10.34727/2022/isbn.978-3-85448-047-1_7. This work is licensed under the 
Creative Commons Attribution-ShareAlike License 4.0 International (CC BY-SA 4.0). 
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tionally, the body was considered as a transparent tool for the realization of composers’ 
intentions that were codified in notated musical works; today, it is valued in its own right 
as the central mediator between matter and mind, between culturally diverse musical 
phenomena and meanings, and between musicians and listeners alike. 

In previous work (Antoniadis, 2013), I have claimed that this focus on embodiment 
takes specific forms of ocularcentric? objectification, which privilege the visual dimen- 
sions of musical performance. I have suggested that the historical development of post- 
WWII discourses on the performing body follows the general scheme proposed by the 
cultural anthropologist Jean-Jacques Courtine (2006): ‘Where once we had subjects 
without bodies, now we find bodies without subjects’ (p. 166). Vivid examples of this oc- 
ularcentric objectification of the body may be found in Stefan Drees’ (2011) overview of 
body discourses in music after 1950, as well as in Harry Lehmann’s (2012) philosophical 


examination of the digital revolution in music. 
According to Drees (2011), 


the body is set on stage not only in terms of the sonic outcome of performative acts, but 
also with regard to its visual aspects as artistically relevant object. This results in the 
conception of the body as a medium [...](p. 13., Translation and italics by the current 
author.) 


Thus, the liberation of the body from the performative restrictions of the past coincides 
with a liberation from the monopoly of disembodied sonic ideals. Through the visual 
perception of bodily actions and images, music becomes an affair of the eyes as much 
as of the ears. For Drees, this implies that musicology can expand to include previously 
neglected genres, such as installations and performance art. This shift from the bodiless 
compositional subjectivity of the past to an audiovisual projection of the musical body 
corresponds to Courtine’s objectification schema. 

Similarly, and with direct reference to the French curator and art critic Nicolas Bour- 
riaud (2002), Lehmann considers this expansion as the emergence of a ‘relational mu- 
sic’. Absolute music is explicitly judged to be irrelevant in a digital culture, and music 
is understood as forging relations to images, performative actions and words, or what 
Lehmann (2012) describes as the strategies of ‘visualization, theatralization, semanti- 
cization’ (Visualisierung , Theatralisierung, Semantisierung) (p. 218). A shift from the 
traditional musicological dichotomy between absolute and programmatic music towards 
a new one, between visible and invisible music, seems to have emerged. 

Moving on towards more systematic approaches to musical embodiment influenced by 
4E cognition (embodied, embedded, enactive and extended), this ocularcentric objectifi- 
cation is manifested in the very theories of embodied gestures. I will refer here to Jense- 
nius et al.’s (2010) overview of relevant literature, as well as to Shaun Gallagher’s ‘in- 
tegrative theory of gesture’ (Gallagher, 2005) and Marc Leman’s communication model 
through corporeal articulations (Leman, 2008). In all these cases, I am interested in 
the fuzziness between the components of musical gestures that are visually conveyable, 
representable and communicable (defined as ‘body image’ by Gallagher), as opposed to 
their components that are irreducible to internal or external representations (defined as 
“body schema’). 


?The term ‘ocularcentric’ is adopted here from Jay (1994). 
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Typically, bodily gesture assumes the role of a link between physical movement and 
meaning, or as Jensenius et al. (2010) put it: 


(..) the notion of gesture somehow blurs the distinction between movement and mean- 
ing. Movement denotes physical displacement of an object in space, whereas meaning 
denotes the mental activation of an experience. The notion of gesture somehow covers 
both aspects and therefore bypasses the Cartesian divide between matter and mind. 


(p. 13) 


To explain musical gesture’s hybrid nature, Jensenius et al. adopt typologies and method- 
ologies that bear unmistakable iconic elements. For example, the distinction between 
communicative, control and metaphoric gesture (Jensenius et al., 2010, p. 14), adopted 
by McNeil’s (2000) approach in linguistics, reveals ocularcentric characteristics that 
permeate these distinctions: Regardless of whether gestures accompany speech to var- 
ious degrees, according to the Kendon continuum (Kendon, 2004) in communicative 
models, whether they are an integral part of computational systems (the control model 
of gesture in human-computer interaction), or they are abstracted as musical concepts 
through metaphor, they are invariably objectified in visual terms. A good example of 
this objectification is offered by Jensenius et al.’s methods of musical gestures’ analy- 
sis, which include interlinked spatial and functional components. Spatial components 
include performance scenes, body positions, and frames of action, similar to Laban’s 
notion of the kinesphere, whereas functional components include the well-known dis- 
tinctions between sound-producing, communicative/expressive, sound-facilitating and 
sound-accompanying gestures (Cadoz, 1988; Dahl et al., 2010). Strikingly, the visual 
analysis of gesture is maladapted to the open-ended, nested, coarticulative nature of dy- 
namic musical gestures developing in multiple temporal planes. 

The aporias cited above (hybrid nature of gesture as bridging the mental and the phys- 
ical and its functional compartmentalization and fragmentation) are addressed in Shaun 
Gallagher’s ‘integrative theory of gesture’, based on a distinction between body image 
and body schema. His theory is developed through the merging of two families of gesture 
theories (motor and communicative) and experimentally confirmed through the study of 
a deafferented subject, Ian Waterman, who despite his lack of proprioception, is able to 
gesticulate even in a blind condition. 


Gallagher (2005) takes an important step away from the ocularcentric constitution of 
gesture, through the distinction between body image and body schema: 


I defined body image as a (sometimes conscious) system of perceptions, attitudes, be- 
liefs, and dispositions pertaining to one’s own body. It can be characterized as involving 
at least three aspects: body percept, body concept, and body affect. Body schema, in 
contrast, is a system of sensory-motor processes that constantly regulate posture and 
movement—processes that function without reflective awareness or the necessity of 
perceptual monitoring. (p. 38) 


This distinction becomes fundamental for his integrative theory of expressive gesture, 
whereby it manifests as a distinction between morphokinetic and topokinetic properties: 
the former are related to linguistic, cognitive, communicative, body image properties, 
whereas the latter are related to proprioception and are controlled by body schema. The 
fact that Ian Waterman can control morphokinetic (but not topokinetic) properties of 
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gesture under non-feedback conditions indicates that expressive gestures rely on a com- 
pletely different mechanism than instrumental or locomotive actions. They are inextri- 
cably linked to communicative and linguistic mechanisms that require no proprioceptive 
or visual guidance, even though they may themselves be visible. As for their topokinetic 
characteristics—the ones that firmly place gestures inside an objectified visual space of 
spatial coordinates—they are controlled by the ‘blind’ processes of body schema. In 
that sense, the relation between gesture and visuality is more convoluted than many tax- 
onomies of gesture indicate. A final touch on the ocularcentric objectification of musical 
embodiment may be found in Marc Leman’s (2008, pp. 160-162) model of musical 
communication between performers and listeners, based on what he terms ‘corporeal ar- 
ticulations’. Corporeal articulations are bodily movements that encode the performer’s 
musical intentions and become transmitted to the listener in the form of biomechanical 
energy through a mediator: the musical instrument. First, the performer’s biomechanical 
energy is transferred to the instrument, a part of it transformed into sound and another 
part bounced back as haptic feedback. This haptic feedback, in combination with sonic 
and visual feedback, creates a closed loop, which is crucial for the performer’s control 
of the instrument and for the illusion of body transparency, the fact that the instrument 
feels like a prosthetic extension proper of the body. Then, the performer transmits the 
sonic and visual energy to the listener, who can decode its meaning through mirror pro- 
cesses, meaning the imitation, explicit or implicit, of the original corporeal articulations 
and her mimetic resonance to them. This model allows for great interpretational lati- 
tude in that corporeal articulations may carry semantic meanings that are different for the 
performer and the listener, but universal in their sensory materiality. Crucially, this ma- 
teriality irreducibly includes the visual modality, which is at least as central as sound in 
the transmission and meaningful decoding of corporeal articulations between performer 
and listener. As for touch, it invariably remains attached to the notion of haptic feedback, 
a fact to be problematized shortly. 


7.3 Covid-19 as a catalyst for the ocularcentric rendition of embodied 
experience into data in cognitive capitalism 


The ocularcentric objectification of the musical body is further illuminated through its 
biopolitical origins and neoliberal mutations, which lead to the political economy de- 
scribed under the rubrics of cognitive capitalism (Moulier Boutang, 2007; Neidich, 2013) 
and surveillance capitalism (Zuboff, 2019), and accelerated through the ongoing sanitary 
crisis of Covid-19. We will describe these developments in reverse order, starting with 
the current crisis and gradually unfolding the wider historical horizon that contains it. 
The imposition of biopolitical lockdowns, curfews and socio-physical distancing mea- 
sures on a global scale since March 2020 has been almost unequivocally justified across 
the political spectrum as an inevitable necessity in order to relieve systematically under- 
funded national health systems under neoliberal regimes during the pandemic stress, and 
to flatten the epidemiological curves to compensate for limited testing and ICU capaci- 
ties. In the context of the ensuing financial meltdown, art forms based on an economy 
of physical presence and performer-spectator co-existence, such as live music perfor- 
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mance and theatre, have entered an existential crisis unprecedented since World War II.’ 
Performing musicians' financial resources are collapsing, due to both the elimination 
of income from live concerts and the reinforcement of streaming services’ domination, 
which was already a factor in the growing precarization of professional musicians prior to 
Covid-19.^ The ‘wet markets’ of wild animals,? epicentres of modern epidemics since the 
end of the 20th century, are revealed to be intimately linked to the “wet markets’ of mu- 
sic: the immoderate and avaricious proximity of humans and animals, as the generator of 
zoonotic diseases? in the context of the wider ecological crisis, threatens inter-animality 
(interanimalité) and inter-corporeality (intercorporéité) (Boccali, 2019 after Michel Fou- 
cault) as the foundation of the musical act. The physical co-existence between musicians 
and listeners/spectators, a *wet market' of impulses and products of the musical body, of 
excited emotions, sometimes of tears, sweat and blood, is now though at the disposal of 
yet another ‘wet market’: the musical body is appropriated, ‘slaughtered’ and turned into 
an object of transaction in the markets of streamed digital data among interconnected 
brains. The French economist Yann Moulier Boutang (2007) defines this configuration 
as ‘wetware’ (brains) and ‘netware’ (network) the biological and social layers comple- 
menting the traditional distinction between software and hardware, in the context of what 
he terms ‘cognitive capitalism’ (p. 89). 

More generally, Moulier Boutang defines cognitive capitalism as the third stage of 
capitalism, after mercantilism and industrial capitalism. Its main feature is the appropri- 
ation and capture of the multiplicity of human experience by digital forms of capitalism. 
This general definition is articulated through fifteen different markers (Moulier Boutang, 
2007, pp. 85-94), which outline the new relationships between advanced information 
and communication technologies, new forms of consumption and production defying the 
patterns of industrial capitalism, and the ubiquitous importance of the appropriation of 
tacit or implicit knowledge, including, for example, the knowledge of how to play an 
instrument. In his own contribution that is more oriented towards the culture industry, 
Warren Neidich (2013) defines four basic characteristics of cognitive capitalism, namely, 
the predominance of knowledge as commodity, the new conflicts between capital and 
immaterial labour, the new forms of computational machinery and a new relationship 
between cultural and neural plasticity (p. 15). It is here explicitly claimed that not only 
does the appropriation of knowledge shape culture and economy, but also that the outputs 
of this interaction feed back to the very constitution of the human nervous system, pro- 
ducing positive externalities such as innovation and seamless human-machine virtuosity, 
as well as negative ones, such as the psychopathologies commonly associated with cog- 
nitive capitalism (lethargy, stress, depression, tunnel vision and burn out). As far as the 
political economy of the arts is concerned, these characteristics of cognitive capitalism 
are usually associated with questions surrounding intellectual property, copyright, open 
access, intangibles, innovation and enterpreneurship—a good example would be the cur- 


Shttps://www.culture.gouv.fr/Sites-thematiques/Etudes-et-statistiques/Publications/Collections-de- 
synthese/Culture-chiffres-2007-2020/L-impact-de-la-crise-du-Covid-19-sur-les-secteurs-culturels 

(access 07.01.2021) 
^https://www.weforum.org/agenda/2020/05/this-is-how-covid-19-is-affecting-the-music-industry/ 

(access 07.01.2021) 

5https://en.wikipedia.org/wiki/Wet. market (access 07.01.2021) 

6A zoonotic disease (or zoonosis) is an infectious disease that has jumped from a non-human animal to humans. 
https://www.who.int/news-room/fact-sheets/detail/zoonoses (access 20.05.2021) 
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rent discussions around the exchange value of streaming in relation to the precarization 
of musicians during Covid-19. Nevertheless, this text aims at a different target, namely 
at the ‘molecular’ level of musical performance as embodied interaction and how these 
political economies do and will shape it in the future. 

This molecular level of performance-related data takes us to the latest twist of cog- 
nitive capitalism, defined by the US social psychologist Shoshana Zuboff (2019) as 
‘surveillance capitalism’. In her research over the last 20 years, Zuboff offers a car- 
tography of the unregulated ‘wild west’ of what she calls ‘body rendition’. The term 
‘body rendition’ expands on the appropriation of human knowledge by the apparatuses 
of cognitive capitalism, as Moulier Boutang and Neidich had already described it. It 
can be defined as the appropriation of embodied manifestations of human behaviour in 
the form of a ‘behavioural surplus’ of interaction data, which are used by the GAFAM’ 
corporations for behavioural prediction, modification and eventually control. As Zuboff 
puts it, “ownership of the new means of behavioral modification eclipses ownership of 
the means of production as the fountainhead of capitalist wealth and power in the twenty- 
first century’ (Zuboff, 2019, p. 18). The human body is re-imagined as a behaving object 
to be appropriated for indexing and research, through a variety of data, which range from 
interaction data in the form of click-rates to GPS location data, movement acceleration 
data and intimate biometric monitoring. 


Before speculating about how such performance-related data may be changing musi- 
cal performance in the near future, it is important to stress musicians’ central contribution 
to the very development of these interactive technologies. Crucially enough, the increas- 
ing virtuosity and performativity required at the user level through the historic develop- 
ment of human-computer interaction, from command lines to graphical user interfaces 
and eventually to tangible user interfaces and forms of augmented or virtual reality today 
(the so-called third wave of human-computer interaction), has been directly influenced 
by music performance. As the Canadian computer scientist and designer William Buxton 
(2008) puts it, 


the real objective of the system’s designers was to study human-computer interaction, 
not to make a music system. The key insight of Ken Pulfer, who spearheaded the 
music project, was that to do this effectively he needed to work with users in some 
rich and potent application domain. And he further realized that music was a perfect 
candidate. Musicians had specialized skills, were highly creative, what they did could 
be generalized to other professions, and perhaps most of all—unlike doctors, lawyers 
and other "serious" professions—they would be willing to do serious work on a flaky 
system at all hours of the day and night. (cited in Holland, 2013, p. 3) 


Just as today’s users interact in clicks and steps or tweets and notifications, perform- 
ing musicians have always been interacting in breaths and beats, cues and signs, and 
the relation between the two is reciprocal, meaning that an increased ‘musicalization’ 
of cognitive capitalism interfaces and an increased rendition of performance data shape 
each other. Given the added fact that the most democratized systems today feature com- 
binations of graphic user interfaces on the internet, it becomes clear that an ocularcentric 
constitution of these musical behavioural data is the norm rather than the exception. 


7 Acronym standing for ‘Google-Amazon-Facebook-Apple-Microsoft’, as used in the relevant bibliography. 
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Before moving on to the variety of musicians’ responses to Covid-19 and how they 
affirm the central role of ocularcentrism in the audiovisual cultures of the new era, it is 
worth stressing the historical background of these developments, namely the fact that the 
objectification of musical embodiment has been a constant in music history through its 
technological mediation: technology has always been performance’s ontology. 

Firstly, one may consider the very biopolitical origins of musical performance itself, 
through the formalization of disciplinary techniques of the body, the increasing institu- 
tionalization of music education, the abstraction and symbolization of the musical act in 
the form of notation, the emergence of the notion of musical work and the strict hier- 
archies between composer and performer. Wolfgang Lessing (2014) has vividly shown 
the shifting meanings of the notion of performing technique, from a disciplinary method 
enforced from the outside to an internalized self-monitoring, a surveillance technique 
of the self, and it is here claimed that such developments foreshadow developments in 
human-computer interaction. Secondly, the history of music technology itself, defined 
by Douglas Keislar (2009, after Marshall McLuhan) as a series of ‘mutilations’ and ‘aug- 
mentations’ of materials and agents, affirms that the shock of Covid-19 is nothing but a 
catalyst for diachronic processes of abstraction and absorption of musical performance in 
the current apparatuses of cognitive and surveillance capitalism: ‘mutilating’ the physi- 
cal co-existence and ‘augmenting’ its digital liquidization, the current crisis extends the 
historical process of an ‘alchemical transformation’, from pure praxis, to symbols, to the 
registration of physical energies and to their final rendition as digital data. Following 
the reflections of the French philosopher Jean-Luc Nancy on the ontology of technology, 
one could consider these diachronic processes of abstraction in musical performance as 
a ‘dehiscence®’: not an opposition to nature, but rather a bifurcation of the organic na- 
ture of musical performance, the creation of a relationship to itself. In this conception, 
the body forms a constant between real and virtual wet markets, whether in terms of an 
animality to be slaughtered or of networked and diffused brains. 


7.4 The last unassimilable frontier in cognitive capitalism: an enactive 
conception of touch 


This double crisis in the economy of the performing arts, which are based on physical 
presence, has provoked a multitude of creative responses, both artistic and technological, 
which fuel speculation about the future of live music. From solidarity concerts impro- 
vised on balconies to the virtual concert halls of established festivals; from the anarchic 
proliferation on social media of concerts by precarious musicians in their most intimate 
private spaces to the near monopoly of the teleconference platform Zoom, which has 
become the new wall of our online communications, including rehearsing and teaching; 
from the struggle against latency in the live transmission of auditory signals to the efforts 


8Dehiscence indicates the spontaneous splitting in plant structures in order to release their contents, such as 
seeds or pollen. Here is the original quote by Nancy: 'La technique ne peut étre opposée à la nature, elle ne 
peut méme se manifester comme dénaturante ou comme destructrice de la nature qu'à partir de sa provenance 
naturelle. .... Cette différence n'est pas une simple distinction de proriétés : elle se présente comme une 
déhiscence, c'est-à-dire comme le décollement interne d'une méme ligne ou d'une méme surface (à la manière 
dont s’ouvrent les anthéres d'une fleur)’ (J.-L. Nancy, 2020, p. 65) 
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for a rapid democratization of new interactive technologies and networked performance, 
various questions arise. What would be musical performance’s destiny in a situation 
of generalized digital mediation of audience-performer interactions? Could a culture of 
physical and social interaction simulations in a virtual hall ever be a substitute for real 
experience? Can we distinguish between physical distancing and social distancing? Be- 
yond telematic performances in the form of ‘Zoom concerts’ with poor image and sound 
quality, what exactly could the potential integration of physical interaction, for example 
movement or haptic data, bring about in the context of the Internet of Things (IoT), or of 
an augmented/virtual reality of the concert? What are the repercussions concerning the 
remuneration of virtual musicians or the copyright for live concerts that remain online? 


My provisional answer to these questions lays bare the ocularcentric constitution of 
musical performance in the Covid-19 era: an audiovisual abstraction of musical perfor- 
mance, through democratized but low-quality machinery of live signal transmission, or 
even high-quality virtual concert halls, in the condition of a diffused Bentham panop- 
ticon? safeguarding a two-way surveillance, both of the musician performing for the 
invisible crowd of solitary eyes and ears, and of the audience, whose metadata are con- 
stantly tracked, indexed, evaluated, deanonymized and sold by the invisible ‘data barons’ 
of GAFAM, remains agnostic as to a dimension that interconnects physical presence, 
intercorporeality, social interaction, sonic vibration, energy circulation and affective po- 
tential. This dimension is touch. 

One should not rush though into simple conclusions as to potential remedies of this 
lacuna of touch in the Covid-19 era, including the integration of haptic interactions in 
the current audiovisual apparatuses or in forms of virtual and augmented reality: the 
notion of touch I am referring to here encompasses a range of phenomena beyond sheer 
tangibility, vibration, or haptic/force feedback design for virtual instruments, as explored 
for example in Papetti and Saitis (2018). 

Drawing on my previous work on piano touch from a continental philosophy and 
radical embodied cognition point of view (Antoniadis, 2021), I attempt a deconstruction 
of the normative perception of touch as physical contact, through its enactive rethinking 
in terms of movement coarticulation, multimodal diffusion, limit experiences and body 
transparency. A final word will be on the relationship between touching and listening, 
which opens these reflections to the communicative, social and deprivatized aspects of 
musical performance. In that sense, touch will be considered as a real, non-metaphorical 
feature that permeates through the communicative chain composer-performer-listener, 
and as a metonymy for a musical ecology, which is invariably physical, mental and social 
(Guattari, 1989). 


From a philosophical point of view, touching has never been a transparent concept. 
Jacques Derrida's main contribution in what is a virtual encyclopedia of the philosophy 
of touching (Derrida, 2000/2005) is the deconstruction of a rudimentary phenomenol- 
ogy: touching is not simply about physical contact with a surface, about tactility, about 
immediacy and presence, about a specific modality, or a specific sensory organ. Touch is 


?The panopticon is a type of institutional building and a system of control designed by the English philosopher 
and social theorist Jeremy Bentham in the 18th century. The concept of the design is to allow all prisoners of 
an institution to be observed by a single security guard, without the inmates being able to tell whether they are 
being watched. https://en.wikipedia.org/wiki/Panopticon 
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rather a cascade of mediations in all senses, which render accessible something untouch- 
able. It is about interruption, tact, discreetness and tangents, as opposed to penetration 
and violence, especially so in the case of palpable effort against the resistance of a limit 
and its non-invasive transgressing. Paraphrasing Aristotle, Derrida (2005) writes, 


(..) but ever since Aristotle suddenly hit on the manifold aporia of touch (aporia, he 
said then, and aporeseie); ever since he, Aristotle, foresaw all the obscurities of the tan- 
gible: touch isn’t clear, ouk estin endelon, he says furthermore; it's adelon, inapparent, 
Obscure, secret, nocturnal. (p. 4) 


He further summarizes the qualities of touch that render it obscure, its Aristotelean 
aporias, as follows: 


= [s touch a single sense or a group of senses? 

=" [f touch is a single sense, what is the organ of touch? 

* [s flesh the organ of touch, or is it the medium, the real organ being inward? 
* [s the subject of touch (haphe, tactility) the equivalent of sound to listening? 


= Are there senses that work from a distance and those that require contact? Or do all 
senses require some form of contact? (p. 5) 


Through this summary, Derrida testifies to the multimodal diffusion of the sense of touch. 
He indeed comes to respond to Aristotle's aporias in the following passage: 


(...) though it is obvious or ‘clear’ [delon] that, first, the ‘organ’ of touch is ‘inward’ 
or internal; second, flesh is but the ‘medium’ of touch; third, ‘touch has for its object 
both what is tangible and what is intangible [tou haptou kai anaptou]’ (ibid., 424a), one 
keeps asking oneself what ‘internal’ signifies, as well as ‘medium’ or ‘intermediary’, 
and above all what an ‘intangible’ accessible to touch is - a still touchable un-touchable. 
(p. 5) 


The issue of an untouchable becoming touchable is inextricably linked to the experience 
of a limit: 


How to touch upon the untouchable? Distributed among an indefinite number of forms 
and figures, this question is precisely the obsession haunting a thinking of touch—or 
thinking as the haunting of touch. We can only touch on a surface, which is to say the 
skin or thin peel of a limit (and the expressions ‘to touch at the limit’, ‘to touch the 
limit’ irresistibly come back as leitmotivs in many of Nancy’s texts that we shall have 
to interpret). But by definition, limit, limit itself, seems deprived of a body. Limit is not 
to be touched and does not touch itself; it does not let itself be touched, and steals away 
at a touch, which either never attains it or trespasses on it forever. (p. 6) 


In other words, according to Derrida, touching has an integrated failure of accessing 
what it actually reaches for, as it by default stops at a non-bodily, non-invaded limit. The 
destruction of this limit, say through penetration or violence, would immediately signal 
the destruction of the very notion of touch. The limit is to be transgressed otherwise.!° 


'0Beyond the current focus on touch, a complete theory of sense-making in musical listening as transgression 
of immediate perception is offered in Reybrouck (2017). In this, it is argued that the surpassing of first-hand 
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Haptic perception has also been a focus of study in radical embodied cognitive sci- 
ence!! through the notion of dynamic touch. In his overview, Anthony Chemero (2009, 
pp. 154-160) has focused on work by Shockley, Carello, and Turvey (2004), who de- 
fine a touch-specific affordance. According to this research, common illusions in the 
perception of objects through touch, such as the size-weight illusion,!* can be addressed 
not through a supposed erroneous computation, or judgment of the object's weight as 
analogous to its size, but through touch-specific information directly accessible in the 
object. To show this, Amazeen and Turvey (1996) experimented with the so-called ten- 
sor objects, which are specially designed objects of identical shape, size and overall 
weight, where the weight is, however, distributed in different parts of the object. The 
different distribution of weights produced different moments of inertia when the subjects 
attempted to wield them, whether having visual contact with them, or even when the ob- 
jects were occluded and the subjects could only feel them. As a result, the researchers 
showed that humans perceive correctly the weight through the object's inertial potential 
as felt on their wrists. The point of Shockley et al. is that this inertial potential, or as 
they call it moveability of an object, is a touch-specific affordance: information available 
in the environment, into which humans can effectively tap through dynamic touch rather 
than through visualizations and representations. 


Having investigated some basic features of touch (its multimodal diffusion, its me- 
diating, transparent and transcendent nature, and its role in dynamic perception through 
active exploration), we will now see how it relates to listening as theorized in embod- 
ied cognition. Setting aside the fact that direct cross-modal correspondences between 
touch and sight have already been documented (Blakemore, 2005) in the context of mir- 
ror neuron research (Gallese, Fadiga, Fogassi and Rizzolatti, 1996; Rizzolatti, Fadiga, 
Gallese and Fogassi, 1996; Rizzolatti, 2002), that is, the listener may be feeling touch 
just because of seeing it, I will rather pursue the ecological idea that touch as action is 
encoded in the different modalities involved in Marc Leman's communication model al- 
ready presented in the first section. In light of Leman's theory, an enactive conception of 
touch allows for its non-metaphorical, energetic transmission to the listener, beyond the 
narrower sense of touch as the performer's haptic feedback. 

First, touch is transmitted in terms of movement coarticulation. The modes of tactile 
contact in musical performance are hardly decouplable from coarticulated bodily move- 
ment and from the design of the instrument as a prosthesis to the performer's body. For 
example, the normative legato touch in piano playing cannot be considered aside from a 
proper synchronization of the several anatomic parts, which allows for a certain pattern 
of energy transmission to the hammers and the dampers of the instrument. In that sense, 
touch is an organic, inextricable part of what Leman calls corporeal articulations. 


multimodal perception involves mediate knowledge based on cognitive and affective categories, spanning a 
continuum between concrete representation and abstract symbolization and involving distinct temporal cate- 
gories (‘in time / outside of time’). 

lI The main feature of radical embodied cognition in relation to (non-radical) embodied cognition is the rejec- 
tion of mental representations and mental computations as explanatory tools. In their place, radical embodied 
cognition employs tools from ecological psychology, describing the interactions of organisms and their envi- 
ronment, and dynamic systems theory, describing the way systems are changing over time. 

Given two objects of equal mass, people (both children and adults) judge the one with a smaller diameter to 
be heavier. For example, they judge a comparatively small pound of lead to be heavier than a comparatively 
large pound of feathers. 
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Second, we defined touch in terms of its multimodal diffusion. Beyond the stimulation 
of tactile mechanoreceptors, the qualities of touch as movement are codified in other 
modalities, predominantly sound and vision. The expanded palette of touch in these 
actions is not only felt by the performer, through the resistance they induce, but also by 
the listener, through the transmission of their multimodal blueprint. 

Finally, touch in the sense of experiences of a limit is transmitted through the lis- 
tener’s empathetic resonance and mimetic interaction. For example, forms of physical 
constraining of the performer require physical effort against the imposed resistances, 
which is literally felt and re-enacted by the listener. Moreover, this social dimension is 
further amplified by the bodily existence of many listeners in the same physical space, 
an idea initially developed by Maurice Merleau-Ponty (1964) under the notion of inter- 
corporeality. In short, the cross-modal qualities of touch, the visual components of the 
related actions and the empathetic resonance to the exerted effort do not require a se- 
mantic representation of the psychophysical resistances, but create the conditions for a 
primordial experiencing by the listener. 


The enactive definition of touch in terms of movement coarticulation, multimodal 
diffusion, limit transgression and body transparency offers a model for energy circula- 
tion in the ‘aesthetics of presence’, as formulated from a theatre studies perspective by 
Erika Fischer-Lichte (2004). In what she calls ‘the soft concept’ of presence, the sheer 
appearance of the phenomenal (as opposed to the script-related, semiotic) body of the 
performer and its coexistence with that of the spectators is a sufficient condition for an 
effect of presence to arise (one may note here the resonance with Merleau-Ponty’s notion 
of intercorporeality cited above). Later on, it is not the sheer bodily existence, but rather 
the surrounding physical space and the spectators’ active attention, which grants the event 
an enhanced quality of presence, what she calls a ‘hard version’ of presence. In the last 
twist of Fischer-Lichte’s argument, a ‘radical concept’ of presence consists in the activity 
of actually sensing the embodied mind in its unity and the production and distribution of 
performative energy to the audience through techniques of the body. She cites the work 
of the Polish theatre director Jerzy Grotowski and the US theatre director Robert Wilson, 
whereby the musical qualities of physical movement enable the primordial experience of 
touch discussed above. 


In Grotowski it was the concurrence of impulse and reaction, in Wilson there were the 
techniques of slow motion, rhythmicization and repetition, which evoke to the specta- 
tors the impression of a specific presence (Gegenwürtigkeit) and enable them to energize 
themselves (Fischer-Lichte, 2004, p. 170. Translation by the current author.) 


More importantly in relation to the audiovisual abstraction of Covid-19, she concludes 
that ‘[a]n aesthetics of the performative is in this sense an aesthetics of presence, not 
of presence effects, an aesthetics of emergence, not of appearance’ (ibid., pp. 171, 175). 
This sort of presence and energy circulation that pertain to an enactive perception of touch 
are crucially not representable and not simulable through the screen-based interfaces of 
cognitive capitalism. 
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7.5 Conclusion: Zoom walls as a modern allegory on the multiple senses 
of touch 


In the previous sections, an overview of the ocularcentric objectification of musical em- 
bodiment was attempted. Starting off with historical and systematic aspects in contem- 
porary musicology, the phenomenon was further situated in the Covid-19 period. This 
current period functions as a catalyst for weaving together several biopolitical threads in 
music performance, ranging from the origins of the body’s objectification in Western art 
music to embodiment’s appropriation through analogue and digital music technologies. 
It was argued that amidst these developments, the sense of touch in music remains an 
unassimilable frontier. Due to its qualities beyond tactility, namely its enactive definition 
in terms of movement coarticulation, multimodal diffusion, limit transgression and body 
transparency, it can hardly be reduced to its visual dimension. In this way, it functions as 
a model for a genuinely anti-visual and anti-representational corporeality in music, which 
remains multimodal, interactive and dynamic, tailored as a diffused complex system of 
energy circulation. 

The equally decentralized panoptica of Zoom walls intensify the current sense of 
touch deprivation in musical performance. Similar to Flemish Renaissance allegories, 
they evoke the sense of touch through its very lack, capitalizing on mediation and rep- 
resentation.? Unlike the conscious limitation of means nurturing creativity though, as 
in the case of these allegories, our networked visual reductionism tends to operate as an 
enforced substitute of embodied experience: rather than attempting to explore ways of 
re-enacting the sheer range of action and energy circulation that define touch beyond tan- 
gibility, these means reveal a lacuna, which paradoxically and profanely renders desirable 
a ‘musical contagion’ in the midst of a pandemic. 

In one of her responses to Covid-19, Catherine Malabou elaborates on the ambiva- 
lence of touch as contagion and its political dimension. Her object of analysis is Giorgio 
Agamben's notion of the contagion as 'a touch that disenchants and returns to use what 
the sacred had separated and petrified’ ^ (Agamben, 2007 in Malabou, 2020, p. 221). In 
this sense, the imposed distancing in musical performance is both an act of consecration 
of touch, a juridico-political abstraction and purification of the musical act in a state of 
exception, as well as the catalyst for a ‘re-contamination’, for the restoration of the pri- 
macy of touch in the musical communicative chain. Malabou’s point is that ‘exception 
cannot function without its aura, that is without the accursed share that constitutes it as 
exceptional. Contagion is transgressive. Instead of repressing it, let’s make transgression 
contagious again’ (p. 226). It is exactly the transgressive nature of an enactive notion of 
touch that becomes palpable through its absence in the current crisis and the promise of 
its return in the world after. 


13A telling example is to be found in Jan van Bijlert’s masterpiece A Courtesan Pulling the Ear of a Cat, 
Allegory of the Sense of Touch. Beyond the central depiction of an act of touch (the playful pulling of the cat’s 
ear by the courtesan), it is rather the future expectation of the cat’s violent reaction (communicated through 
its angry facial expression), as well as the suggestive nudity of the courtesan’s back (potentially triggering the 
fantasy of a tender, sexualized touch), that communicate exclusively through the visual channel a complex 
experience, essentially multimodal and dynamic. The depiction of touch transgresses the painted surface and 
tells another story (an allegory, from Greek allos=another and agoria=story-telling) with potential moralistic 
overtones. https://eclecticlight.co/2017/04/15/painting-the-impossible-touch/ (access 07.01.2021) 

14 Agamben (2007), Profanations, Brooklyn: Zone Books 
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PART Ill 


EMBODIED GESTURE 


#8 

EMBODIED GESTURES: SCULPTING 
SONIC EXPRESSION INTO MUSICAL 
ARTIFACTS 


ENRIQUE TOMAS, THOMAS GORBACH, HILDA TELLIOGLU 
AND MARTIN KALTENBRUNNER 


8.1 Introduction: Sonic gestures and acousmatic music 


The cultural context of the *Embodied Gestures'! ? artistic research project is acousmatic 
music. Within this musical field, sonic artworks are commonly described through ‘ges- 
tures'. For instance, a composer could assert that a musical passage was produced from 
the combination of sound gestures (Paine, 2004). The concept of 'sound gesture' is tied 
to the aural perception of sonic dynamics. The changing characteristics of a sound event 
during a period of time can be perceived as a trace, as a gesture (Van Nort, 2009). 
Composers in acousmatic music (Schaeffer, 1966; Smalley, 1997; Vande Gorne, 2018) 
have described the tendency that listeners exhibit to deduce gestural activity from sound 
material. They observed how perceived temporal changes in sound materials—often 
called sonic morphologies—would always refer back to sensorimotor sensation. This 
particular effect has offered acousmatic composers a creative playground for exploring 
musical inventiveness, creating suggestive mental images, sonic sensations and associa- 
tions. Interestingly, these observations are compatible with the experimental findings in 
embodied music cognition. In the embodied mind theory (Noé, 2004), perception is not 
something we receive. It is something we actively do. During the action-in-perception 
loop, external stimuli would be incorporated as mental simulations, as reenactments of 


! This article is an extended and revised version of the paper Embodied Gestures: Sculpting Sonic Expression 
into Musical Artifacts previously published by the authors at the International Conference for New Interfaces 
for Musical Expression NIME 2021, Shanghai. This article extends the sections Interface Design, Musical 
Outcomes and Discussion providing further documentation. 

?The Embodied Gestures project, by Enrique Tomás, Thomas Gorbach, Hilda Tellioğlu and Martin Kaltenbrun- 
ner, was funded by the artistic research programme PEEK of the Austrian research funds FWF (PEEK-AR399). 


https:/doi.org/10.34727/2022/isbn.978-3-85448-047-1.8. This work is licensed under the 
Creative Commons Attribution-ShareAlike License 4.0 International (CC BY-SA 4.0). 
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what we perceive. It is particularly important that these simulations can involve sen- 
sorimotor activations. For instance, neuroscientists (Haueisen & Knósche, 2001) have 
observed how pianists activate their motor cortex when they only listen to piano music. 
If auditory perception can also be tied to sensorimotor sensations, a natural explanation 
for the perception of ‘sonic gestures’ would be the inherent production of sensorimotor 
simulations. This is probably why it is so natural to describe sound morphologies (e.g. 
temporal changes in pitch, volume and timbre) as physical activity, as movements. 


How do acousmatic composers practically deal with the sonic gesture notion? Annette 
Vande Gorne developed a theory of energy-motion models building on previous work 
by Schaeffer, Bayle and Reibel (Vande Gorne, 2018). According to her theory, energy- 
motion models are motion archetypes inspired by natural actions like oscillation, friction, 
flux, pressure, etc. For Vande Gorne, the application of these energy-motion models must 
begin at the very early stages of the musical piece’s conception. Composers should devise 
sound materials following a well-defined energy-motion model. For instance, during a 
recording session, the composer first chooses the model and then performs the ‘sounding 
body’ (e.g. objects or musical instruments) having this model in mind?. The objective of 
this process is the production of expressive gestural sound materials for an acousmatic 
composition. Citing Anderson (2011), 


through the energy model, the composer can develop a voluntary awareness of the in- 
ternal stimulus which motivates and governs the energy flow unfolded through physical 
movement that results in gesture. Gesture would be articulated by and at the service of 
a particular energy model. 


Vande Gorne methodically identified the following energy-motion models: percussion- 
resonance, friction, accumulation of corpuscles, oscillation, swaying/swinging, rebound, 
flux, pressure-deformation/flexion, swirls, rotations and spiral. 


Another relevant framework especially conceived to describe sonic gestures is Denis 
Smalley’s (1997) “spectromorphology’. In electronic music, audio processing can re- 
sult in sound materials displaying remote relationships to any known sound-producing 
source. For instance, a recorded human voice digitally processed through convolution 
can be morphed into a radically different sound. Addressing this issue, Smalley pro- 
posed a framework to describe the rich variety of sonic content in electroacoustic music. 
He called it ‘spectromorphology’, as it would consist of a set of tools for 'understanding 
structural relations and behaviours as experienced in the temporal flux of [electroacous- 
tic] music’ (1997). Within this framework, the spectromorphology of a musical piece (i.e. 
temporal spectral flux of music) is mostly discussed in relation to ‘gesture’. For Smal- 
ley gesture is an energy-motion trajectory creating spectromorphological life. Smalley 
specifically describes how listeners always tend to deduce gestural activity from sound 
and introduces the notion of ‘gestural surrogacy’, a scale of relationships between sound 
material and a known gestural model (e.g. first, second or third order and remote surro- 
gacy). As we have seen, the notion of gesture is central in two of the most influential 
frameworks for composing and analysing acousmatic music. In the following section we 
will discuss ‘sound gesture’ from the perspective of embodied music cognition. 


3For a better understanding of the concept of ‘sounding body’ we refer the reader to Thomas Gorbach’s inter- 
view to Annette Vande Gorne in this book. 
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8.2 Sonic gestures, artefacts and embodied cognition 


Many scholars have studied the multi-modal gestural images created by auditory infor- 
mation. The fundamental question of these studies has been elucidating what kind of 
physical gestures listeners associate with various musical sounds. The central hypothesis 
of these studies is that sound affords a form of memory recall to cultural and physical 
referents that themselves afford certain kinds of actions. One major problem in West- 
ern musical thought is the lack of an appropriate apparatus for describing holistically 
experienced musical sound. For this reason, researchers have often employed graphic 
methods, which facilitate the characterization of aural experiences. For instance, Godgy 
recorded participants drawing spontaneous gestures to various musical excerpts (Godøy, 
2008). In this case the intention was studying the gestural rendering of what participants 
just heard. Musical experiences can be very complex and densely packed with events. 
With this experiment Godgy showed divergent results for steady pitch sounds with tim- 
bral changes, with some listeners drawing just a straight line and others drawing various 
curved lines. Indeed, some listeners expressed frustrations when they were asked to draw 
multi-dimensional sketches of what they experienced. 

Caramiaux and Susini (2011) studied causal perception through movement. In par- 
ticular, they tracked people's movements while listening to identifiable environmental 
sounds. Their results indicate that when the causing action is highly identifiable, partici- 
pants mainly mimed the sound-producing action. When no clear action could be associ- 
ated to a sound, participants traced contours related to sound acoustic features (e.g. pitch, 
volume, density, timbre, etc.). These dynamic features are typically called the temporal 
morphology of a sound. 

There are also studies conducted towards understanding bodily gesture during sound 
production. In particular, Godøy, Haga, and Jensenius have developed experiments for 
analysing how people move while they mime musical control (Godøy et al., 2006). This 
gestural ‘mimicry’ has been described as performing ‘air instruments’, or making sound- 
producing gestures without making physical contact with any instrument or object. Ny- 
moen developed a study for tracking participants’ hands while they played ‘air instru- 
ments’ (Nymoen et al., 2011). He showed that the most significant parameters mapped 
by participants were pitch, frequency centroid and amplitude dynamics (volume). 

Due to the lack of language for describing sonic events, participants often showed a 
tendency to look for association in order to understand sound. Users tried to describe 
sound examples in terms of familiar objects. Even in the absence of an object, they de- 
scribed sound in terms of artefacts. Tanaka (2012) asserts that the cognitive mappings 
enacted during these types of studies are always informed, mediated and inspired by 
the actual materiality of the controller used (i.e. size, material, shape, acoustic proper- 
ties, etc.) According to Clarke we all have some ecological knowledge on how sound- 
producing actions relate to sound (Clarke, 2005). As Caramiaux has shown, musical 
cognition is always situated and sonic memories allude to certain objects to explain in- 
teraction. In sum, during the spontaneous rendering of movement people also envision 
artefacts (Caramiaux et al, 2015). 

In conclusion, all these examples illustrate a clear tendency: humans tend to deduce 
gestural and sensorimotor activity from sonic gesture. In other words, we are inclined to 
determine the sound-producing gestures from what we are hearing and the possible ob- 
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jects producing this sound. Sound perception would also be referred back to some type 
of material or artefact. This tendency became the working hypothesis of the ‘Embodied 
Gestures’ project. With the aim of exploring a new possible paradigm for interface de- 
sign, our research builds on the parallel investigation of embodied music cognition theory 
and the praxis of acousmatic music. 


8.3 The Embodied Gestures Project 


We argue that a fruitful path for approaching musical interface design—especially to- 
wards the creation of gestural music—could be the incorporation of archetypal sonic 
gestures as design patterns. Musical interfaces following this design paradigm would af- 
ford the same type of physical gestures that a sound material inspires when it is listened 
to. Our hypothesis is that such interfaces would be especially suitable to emphasize a per- 
former’s gestural embodiment within an instrument. For instance, for performing a sound 
passage made from the circulation of elastic sonic movements in space, we would design 
musical interfaces affording by themselves, and through their physical affordances, sim- 
ilar ‘elastic’ physical gestures to their performers. 

The crucial question at the outset of this project dealt with finding successful ways of 
shaping the affordances of specific objects for suggesting particular body gestures. First, 
it was necessary to understand how listeners spontaneously envision sound-producing 
actions and physical materials from specific sound morphologies. After gaining this 
knowledge, we could then develop a number of interface designs. For this reason we 
planned a methodology based on user-studies and experiential evaluation which could 
help us identify suitable solutions according to design patterns. In particular: 


1. A large size user-study to understand how listeners envision sound-producing ac- 
tions and physical materials while they try to mime control of gestural acousmatic 
music. 


2. A second phase informed by the previous user-study where we would design and 
build digital instruments emphasizing a number of energy-motion models. 


3. Practice-based evaluation through the commission of musical performances and 
compositions to external collaborators. 


8.4 Ideating interfaces from spontaneous cognitive mappings 


In the early phases of this research project, we planned a study on ‘gestural mimicry’ es- 
pecially designed to emphasize the material aspects of listening experiences. The aim of 
this user-study was to understand how people envision and materialize their own sound- 
producing gestures into physical characteristics when designing musical interfaces. Our 
hypothesis was that if participants are asked to mime sound-producing gestures while 
they listen to acousmatic music examples, they would also envision artefacts to produce 
that music. If we were able to find a quick way to mock up those envisioned objects, 
we could collect a repertoire of compatible sound-producing actions and artefacts for 
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particular sonic gestures. This information would inform the subsequent phases of our 
project. 


In the user-study that we originally created, we asked participants to produce physical 
mock-ups of musical interfaces directly after miming control of short acousmatic music 
pieces. We composed five sonic examples in the form of short acousmatic compositions 
emphasizing one of the following motion energies: oscillation/rotation, granularity, at- 
tack/repetition/resonance, friction and pressure.* 


©) Verbal description of sonic gestures 
Sonic — = Actions 
Gestures )) 4) 3 EZ e. 


©) Miming sound-producing gesture 


. Envision of 
Sonic 9 compatible 
Gestures ) = sound-producing 
movements, physical 


forms and objects 


©) Production of mock-ups with clay 
> €?) - New musical interface designs 
- Newt f interacti 
[I Selection of frm and material 


©) Interview & mock-up explanation 


It serves us to understand the 
subjective cognitive mappings 
(bodily movement and materi- 
al/object associations) enacted 
by each of the participants 


Figure 8.1: Resume of the Embodied Gestures user-study 


"These sound examples can be accessed from our website: https://tamlab.ufg.at/blog/embodied-gestures- 
methodology (accessed: 1/12/2021) 
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As illustrated in figure 8.1, the user-study begins with a warming-up session where 
participants verbalized material aspects of the sonic examples (of each of the five short 
compositions). In particular, participants filled online forms to outline the gestures 
they perceived and the possible properties of the physical materials used to produce the 
sounds. After this phase, they were invited to stand up and mime control over the com- 
positions. Directly after this, they spent approximately ten minutes producing a mock-up 
with clay. Once they finished it, they were video interviewed to obtain an explanation 
about their cognitive processes and the objects they envisioned (Figure 8.2). This process 
was repeated at least four times with four or five musical examples. 


Figure 8.2: Participants of the Embodied Gestures study miming control of acousmatic 
music and explaining the mock-ups produced (Photo: Enrique Tomás, 2018, CC BY) 


During four workshop sessions, 60 participants from five different creative back- 
grounds (music, arts, dance, computer science and administration) modelled more than 
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Rotation 


Iterative 


Friction 


Granular 


Figure 8.3: Examples of mock-ups produced for four different energy-motion models 
(Tomás & Gorbach, 2019) (Photo: Enrique Tomás, 2019, CC BY) 


200 physical artefacts (see examples in Figure 8.3). Participants were filmed and in- 
terviewed for the later analysis of their particular multimodal associations about music. 
Participants were divided in groups of three to five persons during the experiment. Each 
session had a duration of approximately two hours. 

From the analysis of our video recordings, we were able to categorize patterns of 
physical gestures produced during the mimicry phase (Table 8.1). 
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Categories 


Actions observed 
per sound example 


Oscillation 


Granular 


Attack- 


Resonance 


Friction 


Attack- 


Pressure 


Malleable 


Pressing 


2,38% 


Stretching 


4,76% 


2,77% 


2,63% 


Bending 


2,T196 


5,2696 


46,87% 


Playing with 
composed 
objects 


Rummaging 


27,7196 


Droping objects 


11,1196 


Digging in 


5,5596 


Breaking 


5,5596 


Touching with 
performer's 
fingers 


Linear 


9,52% 


5,55% 


Circular 


14,28 % 


Free 


9,52% 


2,T196 


3,1296 


Scratching with 
objets 


One hand 
linear 


19,04% 


3,12% 


One hand 


circular 


16,66 % 


19,44% 


5,55% 


52,63% 


Between two hands 


5,55% 


2,T196 


31,57% 


Free 


2,38% 


Mechanisms 


Cranks and wheels 


7,14% 


5,55% 


2,63% 


Spinning 


2,38% 


2,T196 


Air pipes 


7,1496 


Water streams 


2,T196 


Buttons 


3,1296 


Sliders 


2,T196 


Colliding 


2,T196 


Hinges 


2,6396 


One object's 


movement 


Balancing 


2,3896 


5,5596 


13,8896 


2,6396 


Shaking 


5,5596 


Rotation around 
the body 


2,3896 


Drumming 


Finger drumming 


19,44% 


9,37% 


Drumming with 
mallets 


41,66% 


34,37% 


Table 8.1: Repertoire of sound-producing actions per energy-motion model observed 
during the Embodied Gestures user-study 


Sound-producing gestures have been well studied by Godøy, Haga, and Jensenius who 
identified two main types (Godøy el al., 2006). First, those human movements made with 
the intention of transferring energy to an instrument (excitatory gestures). Second, those 
human movements made with the intention of modifying the resonant features of the 
instrument (modulatory gestures). 

From the results, we can say that participants firstly (and quickly) envisioned exci- 
tatory gestures. In the great majority of the cases, it took them less than ten seconds 


IDEATING INTERFACES FROM SPONTANEOUS COGNITIVE MAPPINGS 89 


to spontaneously find a compatible sound-producing gesture for the sonic gesture they 
were listening to. Interestingly, participants easily engaged their bodies into various pos- 
sible actions and internally evaluated whether these movements were compatible with 
the sonic gestures they were listening to. During a second phase, approximately twenty 
to thirty seconds later, and only after self-confirming the central sound-producing action, 
participants introduced additional features to their repertoire of movements. Each par- 
ticipant added other necessary bodily movements to perform the sonic transformations 
present in the music (i.e. progressive changes in pitch, volume, timbre). For instance, 
frequency changes were accommodated by producing the excitatory gesture at different 
heights. Volume was often controlled by modulating the speed of the sound-producing 
gesture. Certainly, this logic would allude to the causal schemes found in traditional mu- 
sical instruments. We also observed the ways sound transformation was engineered in 
the artefacts that participants envisioned. Usually, they added an additional or comple- 
mentary affordance to the initial form or configuration of the artefact they imagined (i.e. 
a new degree of freedom to the object) like knobs, sliders, buttons, additional sensors, 
acoustic effects, change of materials, etc. 


Figure 8.4: Embodied Gestures interfaces produced (Tomás & Gorbach, 2021). Note: 
oscillation and granularity (top), friction and flexion (bottom) (Photo: Elisa Unger, 
2020, CC BY) 
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8.5 Interface design 


The following phase in our artistic research project was centred around designing and 
building musical interfaces emphasizing four sonic gestures used during the user-study: 
oscillation, granularity, friction and flexion. From the knowledge gained with the user- 
study, we proposed a different solution pattern for each of the sonic gestures present in 
the music. These solution patterns are compiled and explained into detail in the table 8.2. 


. Sound-producing action and Gesture modulation and 
Energy-motion Model REA : DE : 
technological implementation technological implementation 
Linear or circular trajectories Pressure in a handheld object 
Oscillation of the hand between two poles and wrist rotation 
Two joint Gametrak controllers Force is measured with 


measuring the distance of the hand | two FSR sensors. 3D wrist 
to the poles as well as its rotation is measured with a 


horizontal and vertical angles BNODO055 orientation sensor 


vs A ; Rotation and vertical 
Stirring objects in a bowl 


Granularity displacement of the bowl 
Contact microphones and Rotation is measured using a 
thresholding electronics measure BNOO55 orientation sensor. 
the activity on the bowl: every Vertical distance is 


impact and vibrating activity over calculated with a 
a threshold VL53L1X ToF sensor 


Pressure effectuated on an object 


as held between the player ‘s hands Not needed 
Friction : . . 
and its rotation around one axis 
A FSR s d 
pressure sensor and a ee 
rotary encoder 
Attack (finger drumming) Flexion of a rigid surface 
Attack + Flexion on a surface (a thin metal plate) 
Large size FSR sensor detecting BNOO0S55 orientation sensor on 
attack and its velocity deformable parts of the surface 


Table 8.2: Solutions adopted for designing Embodied Gestures interfaces 


From these solution patterns, we built various technical versions during the project. 
The visual aspect of the interfaces produced in 2020 are shown in Figure 8.4. 


The interfaces’ technical core is an Espressif ESP32 WROOM microprocessor. It 
captures data from sensors and transmits this information wirelessly to a host using the 
Open Sound Control protocol. In our implementation, the host is always in charge of 
defining a sound synthesis strategy. 


5 Although a fifth sonic gesture, (attack and resonance) was used in the user-study we did not implement it 


MUSICAL OUTCOMES 91 


Figure 8.5: Theodoros Lotis performing Voices with the friction interface (Photo: Elisa 
Unger, 2020, CC BY) 


8.6 Musical outcomes 


To evaluate our project, we commissioned three musical works for ‘Embodied Ges- 
tures’ instruments. The first work was commissioned to the composer and performer 
Theodoros Lotis. In parallel, Jaime Reis composed an acousmatic piece with our in- 
struments. Finally, the ensemble Steel Girls (Angélica Castelló, Tobias Leibetseder and 
Astrid Schwarz) prepared an improvisation for three instruments. Additionally, two of 
the authors (Enrique Tomás and Thomas Gorbach) produced two improvisations for two 
of the instruments. All of these works were premiered and performed on various occa- 
sions in festivals in Austria and Greece. 

We contacted these artists eighteen months before their respective premiere concerts. 
After a one-week training workshop, the artists worked independently for more than six 
months with copies of the four musical interfaces. Some captures of these musical works 
can be observed in figures 8.5 and 8.6. 


8.6.1 Voices: composition and live performance by Theodoros Lotis 


Theodoros Lotis composed and performed a musical work for one friction interface and 
interactive music system. Most of the sound material in Voices (Figure 8.5) consists of 
recordings of syllables and phonemes of an invented proto-language and audio recordings 


For a complete review of this musical piece, we refer the reader to the chapter Gestural and Textural Ap- 
proaches in Composition and Performance with the Embodied Gestures Instruments in this book 
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of dancers’ movements. Lotis studied the friction interface and introduced a taxonomy 
of possible trajectories in what he called the gesture-field, the spatial limits of the energy- 
motion model. They are illustrated in figure 8.6. 


1 2 3 
P 
M 
"d Ad 
0 
RL RR 
4 5 6 


Figure 8.6: Taxonomy of motion profiles and gesture fields as they were used by 
Theodoros Lotis in Voices (Tomás & Gorbach, 2021) (Photo: Theodoros Lotis, 2020, 
CC BY-NC 4.0) 


Theodoros Lotis explains that the gestural typology in Voices does not seek to di- 
vide time into small or larger linear temporal structures but rather to establish a style of 
floating narration. 

The accompanying interactive sonic system in Voices consists of a Markov Chain 
model which stochastically selects the sound contents to be played. The interface's ro- 
tation and pressure values are sent to a mapping network application where they are 
weighted. This strategy is intended to mimic the overlapping one-to-many and many-to- 
one gesture-to-sound mappings found on acoustic musical instruments. 


8.6.2 Magistri Mei - Bruckner: composition for fixed media by Jaime 
Reis 
The composer Jaime Reis explains the origins of this work: 


‘I had this idea for ages to think about polyphony of gesture and space, and then to 
actually have a lot of layers and polyphony and so on. This is one of the conversations 
that I so often had with Annette [Vande Gorne] which is, what are the limits of space 
lines? How many movements can you listen to at the same time?’ 
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Magistri Mei - Bruckner is a sixteen channel acousmatic composition. Interested in 
exploring Anton Bruckner's sonorities and polyphony, Jaime Reis extensively used our 
interfaces to generate sound materials for this composition. In particular, following the 
acousmatic compositional method, Reis recorded many hours with a particular sounding 
body: our interfaces sculpting the sound of a number of GRM audio players loaded with 
a recording of Bruckner's Missa solemnis. After this, Reis worked on the organization 
of the recorded sound materials and on a complex spatialization strategy inspired by 
Bruckner's idiosyncratic use of polyphony. 


For Reis, the process of sound material generation was comparable to the ones he 
usually develops with acoustic instruments and objects. However, he described the dif- 
ficulties he found in defining 3D spatial trajectories with our interfaces. Reis usually 
elaborates them in a highly parametric way, calculating complex spatial trajectories on 
the computer. Reis would have required the development of a dedicated computer appli- 
cation able to map his movements to complex 3D spatial trajectories. 


8.6.3 Improvisation for Embodied Gestures instruments by Steel Girls 


The Steel Girls is an electroacoustic improvisation group formed by Angélica Castelló, 
Astrid Schwarz and Tobias Leibetseder. With a long experience in the scene, the Steel 
Girls members show a clear physical and acoustic approach to improvisation as they 
usually perform with amplified objects. In this case, our interest lay in evaluating how 
our interfaces could be used by a small ensemble of improvisers. 


Figure 8.7: Steel Girls performing with oscillation, granularity and flexion interfaces 
during the Embodied Gestures premiere concert (Photo: Elisa Unger, 2020, CC BY) 


The Steel Girls prepared an improvisation for three of our interfaces: oscillation, gran- 
ular and bending (Figure 8.7). Castelló controlled the oscillation interface and mapped 
the handheld device movement to a tape speed effect resulting in a typical sound-scratching 
effect. The device's angle and orientation were mapped to the central frequency of a 
number of resonant filters. Leibetseder performed a bending interface for controlling six 
parameters of a granular synthesiser. Schwarz played the granular bowlin order to trigger 
and transform the pitch of cascades of short sound recordings (100 milliseconds approx- 
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imately) which were previously taken from the same bowl. Their improvisation resulted 
in a brilliant exercise of musicianship and communication on stage. As they did not plan 
anything apart from how to begin their performance, each member of the trio explored 
the different dynamic ranges of gestures afforded by the interfaces. Angélica Castelló, 
who usually does not perform with digital instruments, asserted before the premiere: 


For me, performing with computers is not sexy, but these instruments, they really are. 
Maybe they will reconcile me with the digital world! 


8.7 Discussion 


What are the main differences between composing or performing with these interfaces or 
with other musical controllers? From the interviews we carried out with our collaborators 
we can consolidate the most important observations: 


1. Gestural mapping: our collaborators explained to us that for elaborating equivalent 


energy-motion profiles or sonic gestures in the past, they usually had to program 
complex routines (in Max, Pure Data, Supercollider, etc.) or they were forced to 
systematically simulate these movements with commercial fader interfaces. With 
the 'Embodied Gestures’ interfaces, gestural ideas are directly embedded into the 
dynamics of the interfaces, in the temporal flux of the movements we perform. 
Therefore, the interfaces directly provide access to these sonic gestures through 
compatible physical gestures. 


. Agency to structure performance: the physical gestures afforded by our interfaces 


highly structure temporal play independently of a composer’s original intention. For 
example, Theodoros Lotis described how these interfaces quickly suggested to him 
the use of ‘loops’, a compositional resource he had never seriously employed before. 
The interfaces afforded back and forth exploration of the same physical movement, 
creating a tendency to explore the space between gestural extremes, which therefore 
resulted in loops. Lotis discovered that his compositional attention focused mostly 
on changing the dynamics of these looping gestures (e.g. duration and amplitude) 
just like another compositional material. 


3. Limited affordances and constraints: Theodoros Lotis explained to us that 


these instruments have limits, and, after the limitless computer, it is good to go back 
to limits. All acoustic instruments are limited, like their tessitura and possibilities 
to articulate sound. And these interfaces have limits too. The way you push, the 
way you move around the objects, dictates how far you go with your time, with your 
temporal structures of music, and with the gestural structures. This was a good thing 
for me. 


The apparent simplicity of our interfaces constituted a meaningful creative con- 
straint for the collaborators. It stabilized crucial aspects of interaction which fos- 
tered musical exploration and inventiveness. 
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4. Tacit knowledge: quoting the words of Tobias Leibetseder, of the Steel Girls ensem- 
ble: 


"These instruments tend to put you immediately in a specific bodily movement, and 
Ilike that because it is like beginning to perform or dance with the instrument with a 
really clear plan. 


In our opinion, these types of interfaces benefit from the extraordinary tacit knowl- 
edge that many performers usually carry. In the case of Tobias Leibetseder, he is 
not only an experienced musician but also a dancer. For a performer who has expe- 
rience in exploring bodily gesture there will be many possibilities for transforming 
musical intention into physical movement, and then finally through these interfaces, 
into expressive synthesized sound. In that regard, we observed how the straightfor- 
ward functionality of our interfaces lowered certain early barriers. No manuals, no 
menus, no special computer music culture is required to operate these interfaces. If 
the devices are well set up and powered, any group of people can benefit from their 
tacit knowledge to create or perform gestural music. 


5. Interpersonal variability: as we have explained, the user-study revealed a great inter- 
personal variability of results. Participants’ mental mappings are highly dependant 
on the person's cultural background, on his or her corporeality and other social fac- 
tors (e.g. temperament, emotional status, etc.). Thus, a pertinent question would 
be whether it is possible to conduct more systematic and broad experimental stud- 
ies collecting data on people's musical gestures and mental mappings, and utilise 
such larger datasets to better model robust inclusive interfaces. Our results indicate 
that, using our design method, it is possible to ideate highly idiomatic interfaces for 
specialized communities of users. However, two different persons will never have 
the exact range of corporeal abilities and cultural contexts (e.g. elderly and disabled 
people). We advocate here for a less language-oriented type of user-centred design 
based on spontaneous bodily mappings; that is, a type of design oriented towards 
what is spontaneously innate and natural in the users's actual sensorimotor system. 


6. More than idiomatic interfaces: not all musicians who compose or perform digital, 
electroacoustic or even acousmatic music are interested in producing music from 
a gestural viewpoint. Therefore, our interfaces could be described not only as id- 
iomatic, but as highly specialized. 


7. Musical aesthetics: Our design paradigm presupposes an interest in sculpting the 
(spectro)morphologies of recorded sound material or lively synthesized sound. If 
the interest of the musician relies on composing within the discrete lattice of pitches, 
rhythms, durations and timbres, the application of our paradigm will probably result 
in a low resolution version of the musical intentions that one could perform with our 
interfaces. 


8. Orchestration: each of our interfaces is specially designed to emphasize only one 
particular sonic gesture or energy-motion profile. In consequence, composers and 
performers may require sets of “Embodied Gestures’ interfaces for composing with 
a diversity of sonic gestures. Although this issue could be understood as a limiting 
factor, we also see it as an opportunity for the creation of interface ensembles. 
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8.8 Conclusions 


Within the field of NIME and HCI we sometimes address complex and overwhelming 
issues. For instance, designing digital systems that enhance a performer’s embodiment 
with the instrument. In this project, we escaped from elaborating complex or intricate 
interfaces. Our methodological approach began with experiencing—as opposed to un- 
derstanding—the idiosyncratic ways of doing in our musical field. In other words, we 
first collected experiential expertise in what performing acousmatic music concerns (e.g. 
user-studies, workshops with composers, studio visits, concerts, building speaker sys- 
tems, etc.). Only then were we able to define what a possible and intuitive solution for 
the issue in question could be. This is what Andrew Koenig called ‘idiomatic design’, 
advocating a solution not only by understanding the nature of the problem but also how 
the solution will be used, taking into account the constraints and cultures hindering its 
implementation (Koenig, 1996). We consider the incorporation of sonic gesture models 
into interface design as an idiomatic solution to the complex issue of disembodiment 
within the field of acousmatic music. This was done, in fact, at the risk of limiting and 
filtering the affordances of the physical artefacts we built. These limitations were per- 
ceived in this case as idiomatic, as creative constraints. However, we are aware that they 
could be evaluated differently from the perspective of other musical genres. Not all mu- 
sicians who compose or perform digital, electroacoustic or even acousmatic music are 
interested in producing music from a gestural viewpoint. For instance, our interfaces 
will not be effective for the production of textural, ambient and drone music. Therefore, 
our interfaces could be described not only as idiomatic, but as highly specialized for the 
production of gestural acousmatic music. Finally, it is important to remark that our inter- 
faces were specially designed to emphasize only one sonic gesture. As a consequence, 
composers and performers may require sets of these interfaces for composing from a 
diversity of models. Although this issue could be understood as a limiting factor, we 
also see it as an opportunity for the creation of interface ensembles, a plausible solution 
towards improving onstage communication between performers of digital music. 
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August 29, 2020 in Ohain, Belgium 


Following her initial classical studies first at the Royal Conservatories of Mons and 
of Brussels and later with Jean Absil, Annette Vande Gorne chanced upon acousmatic 
music during a training course in France. She became convinced by the quality of the 
compositional work developed by François Bayle and Pierre Henry and the revolution- 
ary nature of this new art form: disruption of perception, renewal of composition through 
spectromorphological writing and listening conduction. Vande Gorne took a few more 
training courses on acousmatic music and then she studied musicology at ULB, Brussels 
and electroacoustic composition with Guy Reibel and Pierre Schaeffer at the Conserva- 
toire National Supérieur in Paris. 

Vande Gorne founded the research group Musiques & Recherches and the Métamorphoses 
d’Orphée studio in Ohain. In 1984 she launched an acousmatic music festival called 
L’Espace du Son in Brussels, assembling a 60-loudspeaker system —an acousmonium— 
derived from the sound projection system designed by François Bayle. She is the editor of 
the musical aesthetics review Lien and Répertoire ÉlectrO-CD (1993; 1997; 1998), a di- 
rectory of electroacoustic works. She founded the composition competition Métamorphoses 
and the spatialized performance competition Espace du Son. She has gradually put to- 
gether Belgium's only documentation centre on this art form, available online at 
electrodoc.musiques-recherches.be 

Annette Vande Gorne was professor of acousmatic composition at the Royal Conser- 
vatory of Liége (1986), Brussels (1987), and Mons (1993). In 1995 she was awarded 
the Prix SABAM Nouvelles formes d'expression musicale (SABAM Prize for New Forms 
of Musical Expression). She still gives numerous spatialized acousmatic music perfor- 
mances, consisting of works both from her own repertoire, and those of other interna- 


https:/doi.org/10.34727/2022/isbn.978-3-85448-047-1 9. This work is licensed under the 
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tional composers. Her current work focuses on the study of energetic and kinesthetic 
archetypes. Nature and the physical world are models for her abstract and expressive 
musical language. She is a passionate researcher of the various relationships created be- 
tween spoken word, sound and meaning through electroacoustic technology. Relevant 
works include the Tao cycle and Ce qu’a vu le vent d’Est, which have renewed elec- 
troacoustic music’s ties with the past. She has also made a few incursions into other art 
forms, including theatre, dance, sculpture, etc. 


Part 1 


Thomas Gorbach (TG): In your “Treatise on Writing Acousmatic Music on Fixed 
Media’ you build on the notion of ‘play-sequence’. What is ‘play-sequence’? 


Annette Vande Gorne (AVG): Play-sequence is the result of playing a “sounding body’ 
or ‘sound body’ with a special category of energy-movements in your head and the adap- 
tation to its surface. One sound body can create different energy-movements. 


TG: I would like to ask you about what you refer to as a ‘sound body’. Can you define 
its role in the setting of a play-sequence and an energy-movement? 


AVG: It is the instrument, or better the surface, to play on. It has no relationship to the 
body, and it can be anything that makes sound. I prefer to reject the notion of ‘instrument’ 
because it is too close to the traditional conception of an instrument. 


TG: Is it possible to call it a sounding object? 


AVG: I reject the term ‘object’. It is a term imported from the phenomenology of 
Husserl and Schaeffer which mostly deals with the perception of the ‘objet sonore’. The 
‘objet sonore’ is tied to our listening perception. 

It is the notion of the ‘archetype’! that interests me more, not the notion of the 
sound body. This idea was initially developed by François Bayle, although the choice 
of archetype comes from Guy Reibel’s practice. The central question is: What can I 
communicate directly to the imagination of the audience so that they can immediately 
recognize the archetype’s category I propose? For instance, falling, flying, rubbing, fric- 
tion, oscillating, etc. In order to make a play-sequence I always have in mind a special 
archetype and then I play the sound body in such a way that I can produce this archetype. 
It is not a question of the physical gesture that I produce with my body. It is about the 
archetype’s category, imported from the physical world, that I choose for the piece. I 
always look for “sound bodies’ that can produce this kind of energy in the best possible 
way. 

However, it is necessary to add a musical layer to the 'play-sequence'. This musical 
layer is what I improvise from my personal experience as an instrumentalist, in my case 
the piano, but always in the kind of energy-movement I have chosen. The choice of an 


! Note: Vande Gorne refers to energy archetypes that can be perceived in sonic events. 
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Figure 9.1: Annette Vande Gorne in her studio performing a play-sequence with a 
‘sound body’ (Photo: Annette Vande Gorne, 2021, CC BY-NC 4.0) 


appropriate musical layer is as important as the energy-movement and the sound body. 
But the play-sequence is not the composition. When I work in the studio exploring the 
recordings of a play-sequence, the miracle is that the musical layer of the play-sequence 
is constantly conserved during the process of composition. This the reason why I always 
start from a play-sequence. 


TG: Let’s talk about the embodied gestures instruments we have built. One of the 
project’s basic ideas was building surfaces that could by their appearance and physical 
design alone afford an idea of how they might sound. Like when we see a violin, we also 
know how it sounds. 


AVG: Yes, although you have put them in a sonic layer, because they do not sound 
by themselves.” You made a step more that is the programming of the microchip, and I 
see that you have searched for a relationship between the acting bodily gesture and the 
sounding result. 


Vande Gorne explains here that, as the instruments are digital, they have to be connected to a synthesizer to 
sound. 


102 THOMAS GORBACH INTERVIEWS ANNETTE VANDE GORNE 


Figure 9.2: Annette Vande Gorne’s objects for producing play-sequences (Photo: An- 
nette Vande Gorne, 2021, CC BY-NC 4.0) 


TG: Talking about the embodied gestures, it would interesting if you could comment 
on the following instrument (Figure 9.3). We designed it with the idea of friction in mind. 
Its activating gestures would be pressing and rotating. Do you observe these energies in 
the surface of the controller? 


AVG: I observe that there are two kinds of pressure. One is friction with few itera- 
tive results and the other is pressure understood as deformation. In the latter, there is 
a change in the sonic spectral compound during the direction of the pressure. Like the 
instrument called the jaw harp. So this instrument could in fact perform two types of 
energies. Although, for the instrument, it would be the same movement. It depends on 
the mapping. 
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Figure 9.3: Embodied Gesture Instrument: The ‘friction’ interface (Photo: Elisa 
Unger, 2021, CC BY-NC 4.0) 


Figure 9.4: Embodied Gesture Instrument: Vessel interface for granular control 
(Photo: Elisa Unger, 2021, CC BY-NC 4.0) 


TG: The following instrument is a vessel (Figure 9.4). Which energy comes to your 
mind? 


AVG: Rotation and Spiral. The difference is that in the case of rotation pitch is constant 
while at a spiral the frequency goes higher when the movement accelerates. 


TG: Rotating could also be possible by displacing the vessel around us... 


AVG: No, then it is oscillation. It does not depend on the surface but on what is inside 
the vessel. Ah, correction, it is not oscillation, because it is possible to produce a balanc- 
ing movement with this sound body. It is difficult to control the balancing. Oscillation is 
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always mechanic, quick and regular. It is always possible to hear two poles but not the 
trajectory. Balancing is not regular and it is possible to hear the trajectory between the 
poles. 


TG: Let’s observe this instrument made of metal (Figure 9.5). What energy-movement 
do you associate with this surface? 


AVG: It is a singing saw and I see in that the energy-movements of flexion and pres- 
sure/deformation. 


Figure 9.5: Embodied Gesture Instrument: The ‘bending’ interface (Photo: Elisa 
Unger, 2021, CC BY-NC 4.0) 


Figure 9.6: Embodied Gesture Instrument: The ‘oscillatory’ (noisy-rotation) interface 
(Photo: Elisa Unger, 2021, CC BY-NC 4.0) 
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TG: Finally, this is the first controller we created (Figure 9.6). 


AVG: This is more complex. It is possible to make friction. It is also possible to use 
the tension and vibration of the string holding the intermediate object. It would also be 
possible to produce rebounding sounds. 


TG: The string is not elastic, so rebound doesn’t seem possible to me. 


AVG: Then friction is the right energy because it is possible to go up and down and 
left and right with the handle in the middle. It is also possible to combine panorama with 
left and right, and pitch with up and down. 


TG: You have experience with tape speed modulation in analogue studios. It is obvi- 
ous to combine left and right movement with the speed of the tape, isn’t it? 


AVG: Yes absolutely. It is possible to use left-right movements to control the speed of 
the tape in correlation with frequency change. 


Part 2 


TG: Do you think these interfaces could be useful for composing in the studio? 


AVG: It depends on which parameter you use during this process. 


TG: Maybe for any special transformation? 


AVG: Yes, for example I could use them for progressively changing the central fre- 
quency of a filter, its ‘Q’ parameter or its spectral envelope. We would be working in 
the sphere of sound transformation. For me, there are four different possible actions in 
the domain of transformation. The first is the “fixed action’. This means that we will 
not change the parameter controlling the transformational tool while the music passes 
through different sound files. In this case the transformational tool acts like a sieve. Prac- 
tically, it usually consists of a fixed preset of parameters with changing sound sources. 

The second way to deal with transformation is what I call the ‘mobile’ mode. It 
means acting in a mobile way over the tools’ parameters. The correlation, also called the 
mapping, between the quality of mobile acting and the final parameters’ values is crucial. 

The third acting mode is called ‘control’. In this case, there always exists an external 
system like an LFO, a ramp or a square signal, which acts on the parameter without any 
relationship. Thus, in this case, it is also possible to hear or perceive the external system. 

And the fourth acting mode, more electroacoustic, is the ‘cross-synthesis’, in which 
one parameter of one sound acts on another parameter of the same or another sound. 
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TG: Do you mean that the amplitude values of one sound can act on the frequency 
values of another sound? 


AVG: Yes, this is most frequently done. But everything is far from gesture in this 
domain. 


TG: Do you think that these [embodied] instruments could be used in a special musical 
setting? In other words, who could become a good user for these instruments? 


AVG: They could be useful to control parameters of cycling musical parts with the 
gestures of my body. I could act livelier. This means that I have to have a previous idea 
of how the result would be, or an expectation in my imagination. This question is outside 
of the typical ‘concrete’ (musical) procedure, because in the ‘concrete’ method I do not 
know the musical result before my acting. With these controllers, and with any controller, 
I have to configure the parameters in relation to the bodily gesture. That means always 
before physical action happens. 


TG: You have also experience in educating children. Would these instruments be 
useful for children interested in electronic music? 


AVG: Yes, I see these instruments as being highly interesting for them. The most im- 
portant thing to consider would be making the mapping process? in an adequate and intel- 
ligent way. However, I have an objection. When we make a play-sequence with physical 
sound bodies, the variety of results tends to be higher. For instance, two similar bells are 
never exactly the same object. If children use headphones to record play-sequences, they 
may hear small differences on what exactly is being played by each instrument. This is 
not a problem related to gesture. It mostly depends on the sound body used for playing 
our gestures. Thus, in the case of using controllers for education, it is crucial to define 
the kind of sounds. When I was asked by the direction of the music conservatory to teach 
6- and 7-year-old children for a period of six years, I refused to teach do, re, mi, fa... . 
Instead, I invented games to record play-sequences with sound bodies. My experience 
was that they need to develop communicative relationships with other children during 
these games. One by one, or with the whole group. 

After these experiences, I developed some other workshops with older children. About 
10 to 14 years old. I observed that they first needed to understand the mechanism—how 
the technology of acousmatic music works. I took my Revox* and some tape, and built 
a studio in the classroom. They did the whole process of recording play-sequences, but 
they also liked to perform sound transformations towards building a result, a structure. 

Finally, I would like to point your attention to the confusion created around the notion 
of ‘gesture’. Bodily gesture and musical gesture are not the same. Musical gesture comes 
from the imagination of conducting sound along time. 


5With mapping process Vande Gorne refers to the mapping sensor data to parameters in the synthesizer. 
"The classic Revox tape recorder: https://en.wikipedia.org/wiki/Revox 


#10 

GESTURAL AND TEXTURAL 
APPROACHES WITH THE EMBODIED 
GESTURES INSTRUMENTS 


'THEODOROS LOTIS 


The old man motioned me in with his right hand with a courtly gesture, saying in excellent 
English, but with a strange intonation. ‘Welcome to my house! Enter freely and of your own 
free will!’ 

—Bram Stoker, Dracula 


10.1 Introduction: gesture and texture 


In all types of traditional instrumental music, sound production is intertwined with the 
gestures and the bodily movements of the performer. Furthermore, the instrument is 
regarded as an extension of the body of the performer (Nijs, Lesaffre, & Leman, 2013; 
Schroeder, 2006; O’Modhrain, 2018). Both instrumental and vocal gestures serve as 
means of expression through the performer’s physical motion and muscular energy. In 
this view, gesture is related to texture and vice versa in an inseparable way within a 
form-and-content relationship. 

The relationships between gesture and texture can be examined under the lens of dif- 
ferent scientific and artistic fields, including mathematics and physics, computer science, 
bio-arts, cognitive psychology, neuroaesthetics, neurobiology, neurophenomenology and 
more. The notions of gesture as forming principle and of texture as spectral behaviour 
are of paramount importance for the appreciation of the ‘live’ element in music perfor- 
mance (Emmerson 2007, 2001, 1994b) and of the interactions that emerge between a 
sounding body and a listening mind. Performative gestures and musical textures are ac- 
tive parts of a broader dynamic process that is related to the understanding of the musical 
act (Zanetti, 2019). Reybrouck (2021, p. 2) examines this process through information- 
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processing models of cognition as well as operations of sense-making and models of 
enactivism and embodiment: 


... that emphasize the self-organizing aspects of cognition as an ongoing process of dy- 
namic interactivity between an organism and its environment (Schiavio et al., 2017). 
Musical sense-making, in this view, is not to be reduced to a conception of musical ex- 
perience as a kind of abstract, decontextualized, and disembodied process as advocated 
by the cognitive approach to music listening and analysis. It should address, on the con- 
trary, the actual lived experience of music, which involves more than internal cognitive 
processing and detached aesthetic appraisals. (Maeder & Reybrouck, 2016)! 


Denis Smalley (1986, p. 83) examines the unfolding of music in time and the process 
of sense-making in terms of gestural and textural interdependency: 


The relationship between gesture and texture is more one of collaboration than antithe- 
sis. Gesture and texture commonly share the structural workload but not always in 
an egalitarian spirit. We may therefore refer to structure as either gesture-carried or 
texture-carried, depending on which is the more dominant partner. 


Thus, à sonic structure that contains intense gestures, frequent onsets and spectral 
transformations is perceived as gesture-driven, whereas a structure with minimal spectro- 
morphological changes is perceived as texture-driven. However, a sonic structure always 
contains both gestural and textural elements in different proportions. 

Smalley's description of interdependency implies that both gesture and texture carry 
complementary information about the source, the identity, the formation and the inter- 
nal characteristics of sonic events. Although Kersten (2015, p. 196) is referring to the 
acoustic array and the musical invariants,” his point may support Smalley’s argument 
that ‘there seems to be a lawful causal relationship between the physical structures of 
sounds... and the stimulation of the auditory system’. For Smalley (1986, 1997), the 
physical structures of sounds can be decomposed into gestural and textural relationships: 
gesture generates spectromorphological and textural expectations and texture reflects its 
gestural origin. What Smalley suggests is that gesture and texture are essential elements 
of the acoustic array that transmits the sound and feeds the auditory perception (a sound- 
receiving system) with information. 

However, the predominance of electronic technological tools in music creation chal- 
lenged the relationship between gestural activity and spectromorphological development. 
Simon Emmerson (2001) argues that 'electricity and electronic technology have allowed 
(even encouraged) the rupture of these relationships of body to object to sound' (p. 194). 
This rupture becomes apparent especially in live or real-time performances where vi- 
sion and optical stimuli play an important role in deciphering the gesture-field? and the 
gesture-to-sound causality. As Smalley (1997) puts it, 


! Further analysis is needed in order to decode the relationships between musical stimuli and the mechanisms 
of active reception by the listener. However, a full discussion of that issue is beyond the scope of this chapter 
which concentrates mainly on the spectromorphological implications of gesture and texture in composition and 
performance. A detailed examination of the topic can be found at Reybrouck, 2021; Kersten, 2017 and 2015; 
Schiavio, 2017; Emmerson, 2007, 2001, 1994a and 1994b. 

2See also Balzano, 1986. 

3 According to the ‘local/field’ distinction (Emmerson, 1994, p. 31), ‘Local controls and functions seek to 
extend (but not to break) the perceived relation of human performer action to sound production. While field 
functions place the results of this activity within a context, a landscape or an environment’. In our case the 
term field is local and refers to the gestural topology of the performer, i.e the area within which the performer 
performs his/her gestures. Hence, gesture-field defines the space in which the performer acts. 
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We should not think of the gesture process only in the one direction of cause-source- 
spectromorphology, but also in reverse—spectromorphology-source-cause. When we 
hear spectromorphologies we detect the humanity behind them by deducing gestural 
activity, referring back through gesture to proprioceptive and psychological experience 
in general... Not only do we listen to the music, but we also decode the human activ- 
ity behind the spectromorphologies through which we automatically gain a wealth of 
psycho-physical information (pp. 113-114) 


Bodily motion and causal gesture underlie all perceived spectromorphologies and relate 
them to their source. Whenever the inherent relationship between gesture and its resulting 
sound diminishes or disappears, the reference to the causality loosens up, granting its 
place to the realm of remote surrogacy.^ As a result, performers and audiences become 
increasingly alienated from purely physical sound production. 

This alienation can be detected at different stages of a live or real-time performance. 
The ‘amplification’ of human gesture, often through new interfaces and disproportion- 
ate or naive mapping procedures, may create distorted and unreal sonic structures. A 
performer in front of a laptop producing gigantic masses of sound by merely pressing 
a button is a common example. ‘The loss of appreciation of human agency within the 
sound world loses our immediate sense of the "live" (Emmerson, 2001, p. 206). Conse- 
quently, the bond between performer, audience and sound perception is moderated, if not 
vanished, and the perspective of cause-source-spectromorphology is utterly blurred. It is 
a holy sacrifice though, an Iphigenian oblation for the winds of a new perspective of dis- 
located experiences (Emmerson, 2001, p. 204). Although Xenakis (1985) was referring 
to a new model of artist-conceptor, his remark suits this new perspective: ‘...an abstract 
approach, free from anecdotes of our senses and habits' (p. 3). 

A suitable compromise is described by F. Richard Moore as control intimacy, a no- 
tion that refers, for example, to minute textural differences caused by tiny alterations of 
embouchure position on a tube or bow pressure on a string. Grand or minimal, a gesture 
is a composite act with multiple impacts on the production of sound, as we will examine 
later. 

For the moment, the old question persists: Shall we try to liberate the sound from 
its source? Shall we let it separate itself from its source and continue its own life in 
new spatial perspectives? Or, shall we hold the sound bounded to its source within the 
limitations dictated by the dynamic range of the performer's own gestural typology and 
the instrument's physicality? A great number of electroacoustic music works, whether 
acousmatic or with live and real-time elements, intersect in the shadow of this bifurca- 
tion. Karlheinz Stockhausen's Kontakte for example, *... presents intricate networks of 
relationships whereby differences between instrumental and electroacoustic practice and 
theory can appear simultaneously to conflict and support each other' (Dack, 1998, p. 86). 


‘ 


4 According to Smalley (1986) remote surrogacy defines a state ‘... where links with surmised causality are 
progressively loosened so that physical cause cannot be deduced and we thus enter the realms of psychological 
interpretation alone’. (pp. 82-83). 

5 ‘For subtle musical control to be possible, an instrument must respond in consistent ways that are well matched 
to the psychophysiological capabilities of highly practiced performers... Control intimacy determines the 
match between the variety of musically desirable sounds produced and the psychophysiological capabilities of 
a practiced performer’ (1988, p. 21). 
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10.2 Voices 


Voices 7 by Theodoros Lotis is a piece for one performer and Embodied Gestures inter- 
face, tape and electronics. Its duration is 10 minutes and 36 seconds. The piece proposes 
a proto-human? linguistic theatre consisting of primordial sonic elements and interpreted 
by voices of instinct reactions, voices that transmit impulsive expression, voices of hu- 
mans, birds and frogs, and voices of fear, surprise, intimacy, intrusion, complaisance, 
ignorance, and caress. Voices uses two main categories of sonic material confronting the 
physical and mental states of mobility versus immobility and corporeality versus asoma- 
tous insubstantiality: 


1. A collection of vocal onomatopoeias and articulatory phones and phonemes? related 
to the archetypal feelings of astonishment, fear, bewilderment, revelation, need, 
quandary, irony, uncertainty or ambivalence that are deeply rooted in the forma- 
tion of the origins of emotional life and to innate feelings or primary affects. These 
onomatopoeias, phones and phonemes should not be regarded as representatives of 
any reality but rather as parts of a hypothetical expressive language uttered by imag- 
inative prototypical humans. The following figure presents an attempt to transcribe 
the prosodic pitch and intonation of some of the phones and the phonetic segments 
in Voices into the International Phonetic Alphabet (IPA):!9 


[ e.he] [ m.he] [m. her] [ha:jo] [ hov'pa:] [hop] 


[noo.hai] [her] [a:'ha:] [ao.o] [ba'hi:] [a'i] 


Figure 10.1: Prosodic characteristics of phonetic segments in Voices 


2. Sounds of dancing bodies submitted to gravity and trapped into their own corpore- 
ality: bodies in motion or stillness, in closed or open spaces, periodically inactive 
or carved by inertial forces. 


6Voices uses as its primal sonic material the voice of the director and musician Giorgos Nikopoulos from 
his film ‘The Ox’ (recordings: Giorgos Gargalas) and dance improvisation recordings by Christina Mertzani, 
Enangelos Poulinas and Evangelia Randou (recordings: Theodoros Lotis). Other sounds used include voice 
(Agnese Banti), violin (Nikolas Anastasiou), clarinet (Esther Lamneck) and percussion (Giorgos Stavridis) 
(recordings: Theodoros Lotis and Demetrios Savva). 

"Voices is commissioned by the artistic research project Embodied Gestures and had its premiere in October 
2020 at the Echoes Around Me Festival in Vienna. Thanks to Thomas Gorbach and Enrique Tomas for initiating 
the commission. A performance of the piece can be accessed from the website https://vimeo.com/561752213 
8The Proto-Human (also Proto-World and Proto-Sapiens) language is the hypothetical genealogical predecessor 
of the current languages. Both term and concept are speculative and rather unpopular in historical linguistics. 
The term Proto-Human refers to a monogenetic linguistic origin of all languages, possibly during the Middle 
Palaeolithic era. For more information, see Ruhlen and Bengtson (1994). 

?Phones and phonemes are phonetic segments. Phonemes are specific to a language while phones are not. 
More information on phonetic segments can be found at Port, 2008; Perreault and Mathew, 2012. 

V'These phones are part of the script of the film ‘The Ox’ by Giorgos Nikopoulos. 
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I consider these two sonic categories as paradigms of expressive articulation of what 
I might call embryology of primary emotions since body and voice are both vessels of 
all instinctive gestural behaviour. In my approach, the term embryology refers to a) the 
utterance of vocal/phonetic segments, and b) the sounds of corporeal movements. I con- 
sider both a) and b) as archetypal, pre-linguistic fertilizers of human communication. 
Beside language and music there are inarticulate sounds, groans, moans, exclamations 
and cries—not related to specific languages or musical styles—that can form musical 
elements. Beside dance there are unsystematic bodily movements that can be organized 
into choreographies. In that respect, the sounds of human utterances and of bodies in 
mobility or immobility included in Voices, focus the attention on the evolutionary strug- 
gle to rise, to move and to communicate. Chaitow, in his foreword to Beach (2010) 
describes these tasks as ‘... adaptative processes involved in our anti-gravity evolution- 
ary struggle to rise from the floor—where sitting, squatting, crawling and wriggling are 
more appropriate—to the upright where standing and walking become possible’ (p. viii). 

Throughout the six scenes or movements of Voices the construction of phraseology, 
including composed and composite objects, rhythmical structures and leitmotifs serves 
the articulation of primal emotions via gestural behaviour and textural evolution. 


10.2.1 Typology of gestures in Voices 


Gesture is often regarded as a motion trajectory from a point A to a point B, an ab- 
stract vehicle that advances the textural content forward (Smalley, 1997; Hirst, 2011). In 
Voices, however, the kinetic behaviour of gestures is of minor importance. It is their nar- 
rative abilities that are substantial. In other words, it is not the teleological character of 
the gesture that prevails but rather its narrative appraisal. Gestures lose their property as 
textural chisels, thus liberating their potential to narrate and to create time as storytellers. 

Gesture encompasses spectromorphological changes in texture by pushing the musi- 
cal narrative to its logical (formal/systematic) and ontological (informal/intuitive) im- 
plications. Logical implications refer to the temporal structuring of the sound, which 
contains the onset or attack (how a sound starts), the continuant (how it continues) and 
the termination (how it ends) (Smalley, 1997, p. 115). Ontological implications refer to 
the grouping of onset-continuant-termination into the perceptual categories of beginning, 
middle and ending. The structural elements of onset, continuant and termination create 
spectral expectations. For example, an onset/attack may be soft or abrupt, sudden or 
gradual, expected or unanticipated. Accordingly, a continuant may have the character of 
statement, transition or prolongation (Smalley, 1997, p. 115). A continuant is always the 
outcome of an onset. A termination is always the outcome of an onset plus the continu- 
ant: the end of the story of a sound. In other words, the continuant happens because of 
the onset, and the termination because of the onset and the continuant. Both continuant 
and termination are contingent on the onset. At a higher structural level, the elements 
of onset, continuant and termination are shaped by a gesture. For example, when a pi- 
ano key is pressed down by a high velocity gesture, the attack of the resulting note will 
be sudden and abrupt, followed by a prolonged continuant and a gradually decreasing 
termination. I refer to these interrelated structural stages as musical narratives. Through- 
out Voices gestural typologies are used in order to shape textures and utter the musical 
narrative. 
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Figure 10.2: Graphic score of Voices (Photo: Theodoros Lotis, 2020, CC BY-NC 4.0) 


The gestural typology in Voices does not seek to divide time into small or larger linear 
temporal structures but rather to establish a style of floating narration. Hints of this 
style are given in the graphic score by the words ‘cut’, ‘silence’, ‘distant’, ‘whisper’, etc. 
(Figure 10.2). Thus, gestures act as timeless formative vehicles that connect the evolution 
of sonic morphologies with their narrative function. 


10.2.2 Typology of textures in Voices 


Most of the sound material in Voices consists of recordings of syllables, phones and ono- 
matopoeias, and movements of dancers’ bodies. The vocal category comprises mostly 
vowels and, therefore, its spectral content is often harmonic with varying intonation. 
The dancers’ recordings are largely of noisy character with eminent attacks and diverse 
dynamic ranges. The textural character of the sonic material, whether grainy, noisy or 
harmonic, is interrelated with its spatial context. Minor or major spectral alterations are 
directly affected by spatial transformations and vice versa. This is especially the case in 
the fourth scene of Voices, where textural variations emerge through spatiomorphological 
modifications. Occasionally, texture provides the setting for gestural activity, a backdrop 
for the vocal and the corporeal sonic material. 


10.3 The C4 


‘The Controller #4’ (C4) (Figure 10.3) is a member of the Embodied Gestures family of 
interfaces. The Embodied Gestures project proposes 


...à new paradigm of interfaces for musical expression especially designed to empha- 
size a performer’s gestural embodiment within an instrument. For that, ‘embodied ges- 
tures’ explores the possibilities of shaping the physical affordances of designed digital 
instruments with the intention of inspiring particular forms of gesturality. Specifically, 
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[the] objective is to study the implications of designing musical interfaces which can 
afford the same type of gesturality that a particular sound inspires.'! 


The C4 interface operates on the basis of two fundamental types of gesture: pressure 
and rotation. It is built around an ESP32 microprocessor which sends wireless OSC 
data captured by its sensors. This information may be used for parametric mapping 
and sound generation at host devices (i.e. a computer). The C4 is a rotation encoder 
with pressure sensing capabilities. It can be used by rotating its handle (generating four 
increments/decrements per step) and pressing it towards the centre of the instrument. 


The ESP32 


The handle 


The 


Figure 10.3: C4 interface prototype (Photo: Theodoros Lotis, 2020, CC BY-NC 4.0) 


The C4 affords the reinstatement of the performative physical activity as the control 
mechanism for spectromorphological evolution. It reestablishes the gesture-to-sound re- 


!! More information about the instruments can be found in the chapter Embodied Gestures: Sculpting Sonic 
Expression Into Musical Artifacts (Tomas & Gorbach) of this book. The instruments are the outcome of a 
collaboration between the Institute of Media Studies, University of Art and Design, Linz and the Institute for 
Technology Assessment & Design, Vienna University of Technology funded by the Austrian Science Fund 
FWF, Programm zur Entwicklung und Erschließung der Künste (PEEK AR99-G24). 

12 Although in technical terms the C4 is an interface, it can also be described as an instrument since it is 
compatible with various performative actions, including different types of gestures. In that sense, the C4 can be 
regarded as both a performance controller that controls sonic parameters and as an instrument that hosts highly 
expressive performative gestures. 
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lationship and the tactile and visual features of the performance. As Smalley (1997) 
indicates, 


Sound-making gesture is concerned with human, physical activity which has spectro- 
morphological consequences: a chain of activity links a cause to a source. A human 
agent produces spectromorphologies via the motion of gesture, using the sense of touch 
or an implement to apply energy to a sounding body. A gesture is therefore an energy- 
motion trajectory which excites the sounding body, creating spectromorphological life. 
From the viewpoint of both agent and watching listener, the musical gesture-process 
is tactile and visual as well as aural. Moreover, it is proprioceptive: that is, it is con- 
cerned with the tension and relaxation of muscles, with effort and resistance. In this 
way sound-making is linked to more comprehensive sensorimotor and psychological 
experience. (p. 111) 


Although the C4 is not the source of the sound itself but rather the tool for its gestu- 
ral articulation, it affords stimulation and control for sensorimotor integration in perfor- 
mance. 

Apart from reassembling *... some of the cause/effects chains which have been broken 
by recording and computer technology’ (Emmerson, 1994, p. 31) by addressing the issue 
of the stationary ‘live’ sound in performance, the C4 re-establishes the proprioceptive 
energy of the performer (tension and relaxation of muscles, effort and resistance) and the 
awareness of physical presence and motion. 


1 2 3 
P 
oe 
0 ^w 
RL RR 
4 5 6 


Figure 10.4: Gestural curves for pressure and rotation with the interface (Tomás & 
Gorbach, 2021) (Photo: Theodoros Lotis, 2020, CC BY-NC 4.0) 


10.3.1 Typology of gestures 


The C4 renders two main models of gestures: pressure and rotation (including variations 
such as swing and rebound). Pressure is a round-trip action model for exerting force be- 
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tween two poles: from a point of equilibrium to a point of maximum pressure and back 
(Vande Gorne, 2017, p. 19). It deals with the evolution and transformation of both gestu- 
ral behaviour and spectral content. Its attributes include velocity, direction, acceleration 
and deceleration. Rotation is an archetypal model due to its cyclic and repetitive charac- 
ter (Vande Gorne, 2017, p. 20). It involves both a motion and a function. As a function, 
rotation can be applied to other types of gestures including pressure (rotational pressure, 
which equals the pressure-release phenomenon). Figure 10.4 demonstrates six curves in 
shape-space corresponding to the evolution of gestures with the C4 in the gesture-field 
absolute space of the performer. Gestural improvisations on the C4 were recorded with 
a camera. The most frequently occurring gestures of the improvisations were examined 
and outlined roughly in the sketches below. 

The sketches in Figure 10.4 describe some of the gesture typologies within the gesture- 
field. Such paradigms include the cochlea (1, 2), the linear (3), the butterfly (4) the free 
(5) and the square (6) typologies as well as the centrifugal and the centripetal tendencies 
of the rotational gestures. The square typology (6) represents discrete, sequential and 
unidirectional gestures. The three axes represent the directions of pressure (P), the left 
direction (RL) and the right direction of rotation (RR). All the gestures of pressure and 
rotation start from the point 0, which indicates the position of balance of the C4. 

The exact timing of gestural activity during performance is outlined in the action score 
of Voices (Figure 10.5). 


2 VOICES Theodoros Lotis 
SCENE 1 
Time oo 0:2 os 0:20 0:22 0:24 029 032 0:37 0:38 0:40 0:44 
C4 pressure | | | | 
[vol] 
= —= => = 
ui um» C7 Dyn n -— Da 
SCENE 2 
oo 98 0:6 0:22 0:27 0:28 0:35 0:41 1:14 1:16 
| | f : | 
U U 
E T -— A^ es 
SCENE 3 
0:0 0:22 0:39 0:44 0:49 0:54 MIT 103 1:06 Uus 121 1:51 
TTT 
> se — — es =, Short fast 
aN £N ÁN ÁN A 5 


Figure 10.5: Page 2 of the action score for scenes 1, 2 and 3 


As indicated in the score, the volume is controlled by pressure and the panoramic by 
rotation. The performative gestures are divided into the following categories: 


* Long gestures with low velocity / fluid. These gestures concern both the pressure 
and the rotation of the handle. They are mainly preoccupied with the control of the 
overall volume and the panoramic. 
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=" Short gestures with high velocity /agitated. They undertake the microstructural 
spectral evolution (as in scene 4 of 'Voices'). They are often preoccupied with 
the articulation of agitated sonic figures and instant shifts in the stereo image. 


= Circuitous gestures. The circuitous category comprises itinerant motions within 
the gesture-field. As the gesture-field is delimited by the hands and the physical 
motion of the performer as well as the motion of the C4’s handle, gestures can 
wander free or predetermined within these limits. Thus, the ‘live’ element can, yet 
again, be anchored firmly in the domain of the physical, and the energy-field can be 
*... associated with the creation and release of [mechanical] tension which, as we 
know, is at the source of the gesture-field’ (Smalley, 1992, p. 528). 


= Loop enforcement / patterns. The cyclic and repetitive character of both rotation 
and pressure enforces the creation of loops and rhythmical patterns. 


10.3.2 Anatomy of a gesture with C4 


^7 1-7 Ww. — y» nm 


Figure 10.6: A gestural paradigm for Voices 


In Figure 10.6 we can observe the representation of the temporal evolution of a hypo- 
thetical gesture produced by pressing and rotating the handle of the C4. The gesture is 
divided into four discrete parts which are also indicated in Figure 10.6: 


(i) Latency of gesture (preparatory phase). This is the opening stage of the gesture. 
It may last for only a few milliseconds. It is better defined as a revived present, a 
moment of restoration of consciousness that is often experienced during intuitive 
improvisations. Between a very recent past (the residue of a previous gesture or a 
fainted sound) and the expectancy of an immediate future (of a new sound or gesture 
to be born) there is a moment of revived present that is identified with the preparation 
of the gesture and lasts as long as the preparation itself. Thus, this momentary and 
often hesitant latency becomes a site of discovery and discourse. 


(i) Body of gesture. The main part of the gesture is articulated by circular, linear, semi- 
linear or sigmoid curves on the C4 (Figure 10.4). 
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(iii) Residue of gesture. This stage concerns the closure of the gesture. It usually pos- 
sesses a circular or semi-circular trajectory and its intention is to conclude the ges- 
ture. 


(iv) Restorative stage of gesture. On many mechanical instruments and controllers with 
a handle, the lever does not always fully return to its original position. Due to con- 
struction restraints the lever often remains within the confines of positive numerical 
values even after its motion is stopped. This drawback requires a subsidiary (restora- 
tive) gesture that is not intended to produce sonic information but to revert the lever 
to its original position. 


10.4 Mapping network 


The mapping of mechanical performative gestures—such as the ones produced by the 
C4 interface—to sonic attributes raises some important questions: 1) Which and how 
many sonic attributes will be influenced by a gesture? 2) With what percentage or weight 
will these attributes be affected? These questions point out a fundamental issue of the 
mapping, which may be called justification of mapping. That is, to what extent can the 
produced sound be justified by the instrument’s gesture and the predetermined mapping? 

Control operations can be complex and must be analysed prior to any mapping. An 
act of control, such as the movement of the C4’s handle, is determined by several param- 
eters, such as the absolute position of the handle, its velocity and inclination, the degree 
of pressure exerted, etc. A prior to mapping analysis can demonstrate which of these 
parameters play a major role and which have minimal or no effect on the sonic attributes. 


10.4.14 Overdetermination 


These observations point towards the phenomenon of overdetermination,'? whereby an 
event is certified by multiple causes, any of which would be sufficient to account for 
it. Consider the following example: A single movement of the arm that moves the bow 
on a string comprises various components, including pressure, velocity, direction, accel- 
eration, deceleration and inclination. That means that for every alteration in the sound 
(pitch, spectral content etc.) multiple parameters join forces in collaboration. This rule 
is considered as absolute. Although only one of these parameters (e.g. pressure) can 
be used—and usually is in one-to-one mapping strategies—for the alteration of a sonic 
attribute (e.g. pitch), this alteration cannot be fully justified by that use. For example, 
the channelled energy (kinetic, mechanical, automatic, robotic or physical) that moves a 
potentiometer is often reduced into a single value that reflects the position of the poten- 
tiometer, ignoring, at the same time, all the other components of its motion such as the 
speed, the acceleration, etc. This omission of cooperative components may be referred to 
as underdetermination. In reality, however, no change can be made without the holistic 
synergy of multiple parameters. In mapping, as with the bow, a number of parameters 


The term overdetermination (Uberdeterminierung) is used by Sigmund Freud in The Interpretation of Dreams 
as a key feature that explains the presence of multiple causes in a dream. 
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that cause even the smallest change in sound should be taken into account. This is done 
by defining weights for each parameter. 

Overdetermination in Voices is treated by an intermediate application called Mapping 
Network'* that determines the rate of each parameter in the production and evolution of 
sonic events. Ellinas (2020) describes the software as follows: ‘Mapping Network is 
an application for making complex MIDI controller-to-software mappings. Aiming to 
mimic the overlapping one-to-many and many-to-one gesture-to-sound mappings found 
on acoustical musical instruments. The interface is designed after the pin matrix pop- 
ularized by hardware synths!>, with the addition of specifying percentages (or weights) 
to each mapping, rather than choosing just whether two parameters are mapped or not. 
Mapping Network has also built in a rate-of-change calculation feature for control pa- 
rameters, enabling the use of motion as an excitation gesture for sound’. 


10.4.2 The C4 and weighting distribution in Voices 


Figure 10.7 illustrates the processing method of triggered sounds in Voices and the weight 
distribution in mapping. 

Initially, the C4’s rotation and pressure lever sends its values to the Mapping Network 
application (2), where they are weighted via a Max patch (1).!6 Part of the sonic material 
in Voices is triggered by a simple Markov model. The Markov model is a Max patch 
which works with a weighted transition table of probabilities. When active, the pres- 
sure lever triggers a series of random numerical values, which enter the Markov patch 
and cause the calculation of weighting values and transition probabilities. According 
to these probability values, different sonic grains with durations between 100 and 300 
milliseconds are triggered from a buffer of audio files. Subsequently, the sonic grains 
are processed by a pitch shifter that randomizes their pitch and by four delay lines in a 
Max4Live device (3). 

Let us analyse the example in Figure 10.7. Mapping Network (2) is divided into 
columns (inputs) and lines (outputs). The first two columns accept values from the pres- 
sure and rotation lever of the C4. The third column (nul) is a bogus input for all residual 
weights." Each line represents a parameter mapped onto the Max4Live device. In Map- 
ping Network each parameter is correlated with a percentage or weight. In the given 


14Mapping Network is an Open Source software developed by Demetrios Aatos Ellinas as part of his bachelor's 
thesis in 2020 at Ionian University. The software is written in JavaScript and its code can be accessed at 
https://github.com/dimitriaatos/mapping-network 

‘Pin matrices were used for patching audio and control signals in synthesizers such as the EMS VCS3, the 
Synthi 100 and the ARP 2500. 

!6Since the C4 is an OSC controller, its communication with the Mapping Network software introduces latency 
and data loss due to the OSC-to-MIDI conversion. This issue is partly addressed by filtering/smoothing the OSC 
data before reaching the Mapping Network software. In my personal experience (after several performances 
of the piece) the amounts of latency and data loss do not constitute a major drawback, and they do not notably 
affect the relationship between the gestures applied to the body of the C4, the audio processing and the resulting 
sound. In cases where the C4 is used wirelessly, the latency between the interface and the computer is even 
greater. 

17A close examination of Figure 10.7 demonstrates the function of the last column, or bogus input. Consider 
the output line of amplitude 1 (Amp 1). It receives a weight of 0.0 (or 0%) from the C4 pressure column and 
a weight of 0.594 (or 59%) from the C4 rotation column (see line 5 at the bottom right corner of Figure 10.7). 
The residual weight of 0.406 (or 41%) remains unused in the third bogus column (nul). 
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Figure 10.7: An example of weight distribution in Voices (Photo: Theodoros Lotis, 
2021, CC BY-NC 4.0) 


example, the overall volume in the Max4Live device and a minimum and a maximum 
time in the Markov model (i.e. how often a probability is calculated or how often a 
sonic grain is triggered) are controlled by pressure and supplied with weights of 100% 
(1.0), 79% (0.79) and 100% (1.0) respectively (green arrows). Likewise, the delay times 
1 to 4 are controlled by rotation and supplied with weights of 78% (0.78), 67% (0.67), 
52% (0.52) and 23% (0.23). The delay amplitudes 1 to 4 are also controlled by rotation 
and supplied with weights of 59% (0.59), 38% (0.38), 31% (0.31) and 19% (0.19) (red 
arrows). 

The numbers given above are determined by a trial-and-error intuitive approach, which 
mimics the ‘bow-on-a-string’ paradigm: by increasing or decreasing the pressure one can 
affect various sonic parameters to different degrees. Each of the six scenes of Voices has 
its own preset of weightings. Different performers of the piece can choose and apply 
different sets of weights for each scene. This method of mapping defines and justifies the 
resulting sound by acknowledging parametric nuances and level sensitivity. In addition, 
it allows for a certain degree of mapping indeterminacy as the synergy of multiple param- 
eters with different weights does not lead to completely predictable results. As in the case 
of the bow, although we know how a sul ponticello—that combines pressure, inclination, 
etc.—will be heard, we can only predict the resulting sound to a certain extent. 
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10.5 Codetta 


Liveness in performance is directly related to the distribution of the performer’s energy 
through the instrument. Hybrid instruments such as the C4 and the Embodied Gestures 
family of interfaces in general attempt to restore the importance of gestural activity and 
the vitality of the performer’s energy-field. For this purpose, any mapping strategy in 
a live or real-time performance should consider the importance of parametric weight- 
ing and the fact that certain sonic attributes are conditioned by multiple parameters, i.e. 
the texture of a violin note is conditioned by the pressure of the bow, its velocity and 
inclination, etc. 

In that respect, taking into account the different rates of influence that each parameter 
imposes on the sound (overdetermination), we allow for a level of control intimacy by 
restoring the body-to-object-to-sound relationship. Emmerson (1994b) concluded his 
paper on the typology of local/field thus: 


The mapping of performer gesture to control function: expression is multidimensional 
hence individual parameter choice and scale may need to be the result of a cluster of 
parameter controls each following a different law: hence the creation of global control 
functions which ‘decide’ more detailed values. (p. 34) 
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#11 
EXPLORING POLYPHONY IN SPATIAL 
PATTERNS IN ACOUSMATIC MUSIC 


JAIME REIS 


11.1 Introduction 


The multiplicity of musical practices in Western art music that arose from the technolog- 
ical developments involving the use of electricity applied to music were particularly ex- 
pressive after the Second World War. Their associated terminologies are varied: musique 
concréte, Elektronische Musik, acousmatique, cinéma pour l'oreille, sonic art, among 
others. For the purpose of this text, their differences will not be tackled, but some of their 
underlying common characteristics, particularly the use of spatial concepts in acousmatic 
listening will be discussed. The decision to focus on acousmatic music concepts derives 
from the fact that most of the mentioned literature and musical concepts are grounded in 
musical traditions where space is a key element to understand this practice. 

The focus on many of Annette Vande Gorne's conceptualizations is not merely for 
the allusion to her work in the present volume, but rather for her centrality to and legacy 
within literature and music creation in the acousmatic music world. 

This text is organized as follows: A theoretical short introduction to the use of space 
as a parameter in acousmatic music and some of the main perceptive features involved 
in such practice, followed by a brief description of acousmatic music techniques, such 
as those presented by Annette Vande Gorne, and their relations to gestures. The final 
section describes how I have been able in my own music compositions to enhance spatial 
polyphony through various techniques. 


https:/doi.org/10.34727/2022/isbn.978-3-85448-047-1 11. This work is licensed under the 
Creative Commons Attribution-ShareAlike License 4.0 International (CC BY-SA 4.0). 
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11.2 Space 


The study of spatial features in sound in general is a very complex task, but it can be 
handled by perspectives that are focused on perception and reception, framed within dis- 
tinct sciences and associated methodologies. The study of Western art music may often 
involve understanding an increasingly differentiated socio-communicative system gener- 
ated by a multiplicity of included musical practices within its frame. Here, rupture and 
novelty take on an important role, often creating a gap between what one can perceive, 
pay attention to or focus on when encountering such musical practices, and what the 
music-makers would think that should have been perceived (Reis, 2015). 

The exploration and development of techniques that tackle spatialization in electroa- 
coustic music has been a part of the associated musical practices since its origins with 
Pierre Schaeffer, Karlheinz Stockhausen, Varése and other pioneers. However, the liter- 
ature greatly excels the fields around the practices of Western art music history. 

A brief review of some of this literature will be mentioned here, accompanied by some 
personal considerations concerning my own perception and musical practice. 

The study of spatial sound features has been addressed in the frame of different sci- 
entific fields such as physics, acoustics, psychoacoustics, psychology, biological basis, 
technological approaches (Blauert, 2001; Opstal, 2016; Rumsey, 2001; Suzuki et al., 
2009; Wollner, 2018), and in other perspectives. While the importance of such stand- 
points is of substantial significance to the study of spatial sound features in music, other 
perspectives based on creativity are nonetheless key to providing new paths for research 
that are based on artistic practices. 

When trying to understand the conceptualizations and practices of the use of spatial 
features in music, one can find many specialized texts, articles, journal issues, etc. that 
describe a personal practice in general or in a specific work that may include technical, 
conceptual and methodological considerations, often deriving from personal research and 
artistic experiences. 

Many of the personal perspectives that derive from a personal compositional practice, 
such as in the works of Chowning (1971) or Stockhausen (1988) among others, can often 
be extrapolated into other practices. Other texts can be considered from the beginning as 
more comprehensive perspectives: 


* from Western music history that will include both instrumental and acousmatic mu- 
sic within different time scopes (Harley, 1994; Zvonar, 2005); 


= in the systematization of spatialization conceptions in acousmatic music (Smalley, 
2007; Vande Gorne, 2018); 


= in the field of soundscape! (Schafer, 1977, 2008; Truax, 1978); 


= within the spheres of technology applied to electroacoustic music: such as Brümmer's 
comprehensive article that covers diffusion methods? and systems and their appli- 


l Terms such as soundscape, acoustic ecology and bioacoustics seem to have convergent meanings. This prac- 
tice alludes to the study of the effects of the acoustic environment on the physical responses or behavioural 
characteristics of those living within it (Truax, 1978). 

?Such as vector base amplitude panning (VBAP), distance-based amplitude panning (DBAP), Wave-Field Syn- 
thesis (WFS). 
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cations (2017); terminologies, technologies and compositional approaches within 
spatialization techniques (Lynch & Sazdov, 2017); the use of peculiar technologies 
such as the ultra-directional sound that one can obtain with a parametric loudspeaker 
array (Reis, 2016); immersive explorations of virtual performance spaces (Wóllner, 
2018, among others). 


Many specialized journals have dedicated issues to related issues, gathering perspec- 
tives from a significant number of authors, such as Barriére & Bennett (1997), Chouvel 
& Solomos (1998), Dhomont (2008), Keislar (2016), (2017), Szendy (1994), and Vande 
Gorne (2011) among others, where one can find seminal texts from authors such as An- 
derson, Barrett, Barlow, Bayle, Boulez, Chion, Couprie, Dhomont, Dufour, Emmerson, 
Ferreyra, Justel, Lotis, Mandolini, Mary, Menezes, Nattiez, Normandeau, Parmegiani, 
Risset, Roads, Schaeffer, Smalley, Teruggi, Truax, Vaggione, Vande Gorne, and so many 
others who have largely contributed to the discussions of the use of space in acousmatic 
music. 

Within the large topics that arose in literature, very often the topic of spatial concep- 
tions that are focused on ideas of tridimensionality, or conceiving beyond the horizontal 
plane (Kendall, 2010), and other terminologies that have been systematized (Lynch & 
Sazdov, 2017). Groundbreaking works and research can be read about the specificities 
of terms such as dome, cupola, spherical, semi-spherical, immersive, 3D, among oth- 
ers, such as Barrett (2016), Brümmer (2017), Kupper (2008), Normandeau (2009), and 
Stockhausen (1971). I find this literature to be particularly important when it comes to 
the domain of spatial polyphony, for the new possibilities that are opened up by such 
systems in comparison to systems that are based in the horizontal plane only. 


11.3 Spatial polyphony 


The development of acousmatic music practice often relates ideas of musical gesture to 
the perception of movements in space. But where can one draw the line between a mass 
of sounds and movements that are perceived as a texture, and the independence of voices 
and spatial patterns? 

The answer to that question may rely on a large number of studies previously men- 
tioned here. However, for the present text, I intend to explain how I view the concept of 
spatial polyphony as presented in the literature connected to acousmatic music, before 
attempting to respond to the question myself based on my own perception and current 
experience as a composer. 


11.3.1 Asa concept and in practice 


Even in the realms of acousmatic music, the concept of spatial polyphony has been used 
with slightly different meanings according to the context in which they were being used. 

Mary explains the importance of polyphonic materials and space referring to present- 
ing a precise role in electroacoustic orchestration for each element (timbre partial) and 
therefore arriving at a polyphonic conception of space that is coherent with the poly- 
phonic conception of the music materials and their inner connections to the position/ 
movement/ panoramic and to the volume/ mass/ depth (2006). 
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Menard discusses his ideas of a (super) polyphony of space through complex diffusion 
systems designed by the composer (2008). 

Ascione (2008) describes his first experiences in 1985 with the creations of 16 track 
realized in G. R.M. mainly confronting differences between conceiving a spatial work for 
stereo or for larger multi-channel pieces. He mentions what is enabled by what he dubbed 
polyphonic spatial composition: more effective adaptation of the space to the musical 
subject thanks to the permanent control of the affectation of sounds in the aerial sphere 
(when compared to the real time spatial diffusion of a stereo piece); the possibility of a 
better evaluation of the plasticity of the work; the ability to make spatial paths within the 
composition, to give more perspectives, to better distribute the masses and forms which 
oppose and respond to each other, and to specify the locations for the listener. 

Merlier (2011) distinguishes the ideas of spatial polyphony and depth of the [sound] 
field, mentioning that both characterize the superposition or spatial obstruction of several 
objects. The depth of the [sound] field is linked to the geometric occupation of space (in 
the sense of depth): a single sound object with a large spatial mass or several distinct 
sound objects distributed in space. As spatial polyphony underlies something more con- 
ceptual—such as the simultaneous perception of several spaces, or of several spatialities, 
or of several discourses of space—we can see that the two terms overlap in part, but are 
not synonymous. 


I here refer to the latest concept of spatial polyphony as presented by Merlier. Hav- 
ing my ear and personal perspective as a reference, sound spatialization acts as a central 
musical parameter, where the distribution of sound within the loudspeakers is related not 
only to different kinds of movements and spatial shapes and paths, but related and asso- 
ciated to energy models; or, to put it in a different way, associated to spectral changes, 
different speeds and other parameters that make the spatial movements (paths) more or 
less recognizable—and even unrecognizable in given moments. My overall goal in spa- 
tial polyphony, is to enable focus in order to distinguish and be able to describe each 
particular sound spatial movement, in the sense of a recognizable pattern or distinct fea- 
ture that one would group as a single entity, inspired in the way described by the Gestalt 
theorists and the perspectives of perceptual psychology as Deutsch elucidates (1980). 


11.3.2 Limits in the perception of spatial polyphony 


One central question when creating and listening to works involving spatial polyphony 
is: How many independence spatial patterns can one listen to simultaneously? 

Ludger Brümmer (2017) establishes a proper link to polyphony in a common sense, 
referring to the examples of listening to a fugue, where up to and including three or four 
voices it is possible to listen to and follow each voice, whereas five or six voices can 
scarcely be perceived in detail. He does, however, mention that polyphonic listening is 
a skill that can be improved through training. Emmanuel Nunes' (1994) work Lichtung 
I features up to six simultaneous spatial paths distributed totally independently between 
eight loudspeakers. Karlheinz Stockhausen would often talk about the importance of 
perception, referring to Cosmic Pulses, from Klang, as the first of his works where he 
couldn't follow each of the individual (also spatial) lines during the whole piece (Siano, 
2013). While in many of Annette Vande Gorne's writings (2008), in which she alludes 
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to other composers perspectives, and to her lectures*, she professes her perceptual expe- 
rience at the time as being limited to four movements or four differentiated geometric 
spaces. Over the years, personal conversations with Briimmer, Nunes, Stockhausen, 
Vande Gorne and many other composers, scientists and audiences inspired me to con- 
sider what my personal limit might be for the maximum number of simultaneous spatial 
paths, and furthermore, to contemplate how I could construct a first musical work that 
would directly address this issue. I realized that if I wish to surpass the three / four dis- 
tinguishable spaces, I would need an immersive space in a dome shape, since I believe 
that the addition of the dimension of height could provide a more accurate perception in 
the multiplicity of spaces. The work Magistri Mei - Bruckner (2020)* intended to have 
up to seven simultaneous perceivable spaces. The perceptive result has yet to be tested 
in more than one studio or concert; nor has it either been discussed with colleagues and 
audiences, or subjected to further testing from other perspectives. 


11.4 Spatial polyphony in my music 


Ihave previously materialized my interest in aspects of spatial polyphony as a composer 
in previous acousmatic works such as: 


= Omniscience is a Collective — part I (2009), where space is used to enhance seman- 
tic aspects of what I term multi linguistic polyphony (Reis, 2011). This work was 
inspired by an idea from Schaeffer's famous 1966 Treatise, where he mentions spa- 
tial localization as a cue for what could be a polyphony of chains of objects (Schaef- 
fer, 2017). The multiple different languages that one can hear simultaneously could 
only be perceived semantically through space location. 


= Jeux de l'Espace (2015), for eight regular loudspeakers, equidistant around the au- 
dience and one directional parametric loudspeaker array to be operated during per- 
formance, requiring an operator to play it following specific instructions on a score 
demonstrating at each moment where to point, what kind of surfaces to point at, or 
‘swipe’ in the direction of the entire audience, or just parts of it, where the gesture in 
each moment is determinant for the spatial perception of the audience (Reis, 2016). 


" Fluxus, pas trop haut dans le ciel (2017), for 16 channels in a dome distribution, 
where sound spatialization acts as a central feature; the distribution of sound relates 
not only to different sorts of movements and spatial shapes, but to spectral changes, 
varying speeds and other features that render the spatial movements (paths) to be 
more or less noticeable, or even unrecognizable. that I’ve called spirals, rotations, 
spectral explosions, points, geometrical shapes, lissajous curves, sound suctions, 
walls of sound, sound swarms, etc. The intention is not to have them being perceived 
as a taxonomy of movements relating to formal features of the macro structure, but 
rather as the energy flows present in the world and in the work (Reis, 2020). 


On several occasions, in particular while organizing concerts and related activities in Portugal (2015 and 2019) 
and while we were giving lectures together in places such as Kyiv (2018), Ohain/Brussels (2019) and Vienna 
(2019). 

^Work commissioned by the Embodied Gestures artistic research project, funded by the Austrian PEEK pro- 
gramme. 
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I recently composed a new acousmatic work that features a strong personal interde- 
pendence between gesture and spatial polyphony. Inspired by Anton Bruckner’s method- 
ology in conveying traditional polyphony, I have recently been developing ideas of spatial 
polyphony that I try to explore not only through traditional polyphonic development, but 
mainly through sound spatialization connected to musical gestures expressed in spatial 
patterns that travel through a dome-shaped sound system, namely in the piece Magistri 
Mei — Bruckner (2020), for 16 channels in a dome distribution, composed within the 
frame of the project Embodied Gestures, using new musical interfaces? developed within 
this research project that were used in the conception and creation of this piece, along- 
side algorithms, regular patterns and gestures that were materialized in sound objects as 
a counterpoint to spatial polyphony. 

Bruckner’s sovereign mastery of counterpoint can be observed both in the predomi- 
nantly polyphonic textures of the first three movements and in the massive fugue with 
chorale which forms the bulk of interrelatedness of the breakthrough provided by the Fi- 
nale of his Fifth Symphony (1876), where the chorale theme ‘breaks through’ at the end 
of the exposition space (Hawkshaw & Jackson, 2001; MacDonald, 2010). The idea of 
space polyphonic breakthroughs in the organization of the layers was decisive in achiev- 
ing the desired result of multiple layers that are here briefly describe in eight procedures 
that were conceived in connection to specific energy models®, leading to what I consider 
to be audibly distinct spatial patterns: 


1. False polyphony’ in patterns (through changes in amplitude, timbre and so forth); 
2. rotations in the lower ring of loudspeakers; 

3. opposite direction rotations in the medium ring of loudspeakers; 

4. internal geometries (triangular and other shapes, mainly in front); 

5. spectral suctions / explosions, usually from the lower to the upper loudspeakers; 
6. simultaneous interpolated actions in the three rings of loudspeakers; 

7. points/localized actions; 

8. spirals, usually from the lower to the upper loudspeakers. 


5Created by Enrique Tomás and Thomas Gorbach. 

$T use the term energy model as taken from Vande Gorne's work, which in turn draws from the work of Pierre 
Schaeffer (descriptive vocabulary of listening), Frangois Bayle (certain concepts defining acousmatic sound), 
and Guy Reibel (play-sequence and the importance of gesture), and that connects to a specific musical universe, 
usually a physical model, working as an archetype (a fundamental concept in conducting acousmatic listening) 
consisting of creating a sequence by applying a musical idea in relationship with the model (Vande Gorne, 
2018). 

TIt is an analogy to the homonymous term that refers to the connection one can make in the frequency range 
giving the illusion of multiple voices when listening to a single melodic instrument, such as in a Telemann 
fantasia. Here the term is used in the sense of having a recognizable spatial pattern with a sub-pattern that can 
either create a localized action or a construct of, for instance, a sub-pattern of a distinct geometry that enables 
us to hear both the original pattern as well as the new one. 
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Although the perceptual features are less accurate when it comes to the discrimination 
of a sound source within height, when compared to our acute sense of space in a horizon- 
tal plane, particularly in front of us, the possibility of having sound patterns that travel 
above the audience allows an important feature of distinction between patterns. 

When the polyphonic density is increased, the enhancement of a layer is usually 
achieved by a gesture. Almost all the sound material was made by the new musical inter- 
faces developed for the aforementioned Embodied Gestures project. I mainly took short 
samples from works by Anton Bruckner and thereby created hundreds of play-sequences 
having in mind energy-movements using the new instruments and thinking about them 
as sound bodies*. I subsequently ruled out? from the work the majority of the created 
play-sequences, as is usual for me when composing However, one of the most notable 
perceptive features that allowed me to create distinguishable layers was the contrast be- 
tween such play-sequences made with these sound bodies, in contrast to others created 
by algorithms or using other simple programable interfaces.!° This contrast made by the 
hand created peculiar gestural spaces!!, to use Smalley's (2007) terminology, that would 
emerge back on the surface and play a role in the memory, connecting a layer that was 
previously presented, which had started to dive into textural sound masses, but would 
again rise up and be more easily distinguishable to the ear in comparison with the other 
play-sequences. The importance of the physicality of the gesture in the sound result and 
its perception has also been tackled by Brümmer (2017) and Vande Gorne (2018) in the 
context of acousmatic music. 

This latest compositional experience allowed me to give light to two personal ques- 
tions: 


* [s the perception of layers related to a musical gesture? 


* [s there a relationship between such musical gesture and a physical gesture? 


The answer for both questions when considering this work is yes. Although there 
are many ways to convey space polyphony, the composition of Magistri Mei - Bruckner 
allowed me to test new personal limits regarding the perception of the simultaneity of 
spatial patterns in conjunction with new interfaces that acted as sound bodies for the 
creation of my own play-sequences and their interweaving connection to their associated 
energy models, with their idiosyncratic perceptual characteristics that allowed for the 
nourishment of a rich space polyphony. 


8To better understand the concepts play-sequences, energy-movements and sound bodies, I highly recommend 
reading Thomas Gorbach's interview with Annette Vande Gorne, as well as reading her trailblazing work 
(Vande Gorne, 2018). 

?Tn the sense that I created many more sounds than the ones used in the final work. 

Many of the algorithms were created in SuperCollider, both in ways that allowed a sound sequence to be 
triggered by command lines, or in lines that were, for instance, controlled by the MouseX / MouseY. 

11 Gestural space—The intimate or personal, source-bonded zone, produced by the energy of causal gesture 
moving through space, as with performer and instrument, or agent and sound-making apparatus (Smalley, 
2007). 
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#12 
USER-CENTRED DESIGN AS A 
MODEL-BASED CO-CREATION PROCESS 


HILDA TELLIOGLU 


12.1 Introduction 


Due to digitization in almost every aspect of our lives, we are dealing more and more with 
contradicting requirements on technology-based services which are provided by compa- 
nies, education institutions, or governmental bodies. Design with users, design for users, 
and voice of the customer techniques (involving the customer in the definition of the 
products or services) become very important in industry, especially user acceptance is 
key. Methodologically, everything is possible, but not everything is successful. We need 
the right approach for a user-centred development of innovative products. To avoid the 
gap between the use and design of systems, sociotechnology can be utilized as a guid- 
ing approach (Emery & Trist, 1960). Based upon the principles of participation at all 
stages of development processes, user-centred methods have proved themselves as very 
useful means of facilitating open and cooperative settings. Co-creation and mutual un- 
derstanding among users and designers make it possible to design and develop successful 
systems that are acceptable to their users, both in their shape and look-and-feel, and in 
their functionality. This is a promising way to create sustainable systems for real use. 
First of all, we have to understand why we should focus on users in design processes 
(Ritter et al., 2014). The answer is very simple: we want to create a system or technology 
that is intended for human use, no matter how much artificial intelligence is helping to 
carry out certain tasks automatically or semi-automatically, usually non-transparently, in 
the background. We want to design and develop an effective, safe, efficient, scalable, 
and—the most important among all requirements—enjoyable and usable system for peo- 
ple, in which users can experience what they know well because they have experienced 


https:/doi.org/10.34727/2022/isbn.978-3-85448-047-1_12. This work is licensed under the 
Creative Commons Attribution-ShareAlike License 4.0 International (CC BY-SA 4.0). 
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similar things so far in their lives, based on that what they can remember. That is why 
we need to understand them, their characteristics, skills, experiences, commonalities, 
differences, and their use context, in which they will perform certain tasks by using the 
system or technology we provide. Understanding users entails specific methodological 
knowledge and skills on behalf of designers, focusing on ways of getting to know the 
potential users of an intended system, and furthermore, of involving them intelligently in 
the design process as much as possible. The process of understanding in order to inform 
the design for users involves the following actions (Ritter et al., 2014, p. 4): 


= knowing how to observe and document what people do, by using appropriate meth- 
ods, 


= understanding why people do what they do, by gaining insights about people's in- 
trinsic and extrinsic motivations, 


" understanding and predicting when people are likely do things, by identifying peo- 
ple's behavioural patterns, 


= understanding how people choose to do things the way they do them, by studying 
the options people have as well as the constraints and resources that are given. 


All these actions require certain knowledge and skills on behalf of designers to establish 
the appropriate methodology, research, and design setting at the right time. User-centred 
design (UCD) helps to achieve not only a better understanding of users but also involving 
them throughout the whole design process. The consideration of human characteristics 
and capabilities as central factors in the design process (Preece et al., 2015) facilitates 
the creation of better accepted and sustainable systems, which are moreover, used. 

Successful systems are the ones that go beyond individuals’ requirements and capabil- 
ities by also explicitly considering the social interactions and environments of their users. 
Here, sociotechnology is the necessary framework to base the design on. To address all 
these aspects in a design process is not easy. We need to know what we have to ask when 
in the course of design projects. For instance, we have to find out who is going to use 
the system or technology and why. What are the goals of users? Are users willing to put 
effort into learning how to use the system or technology? How often are they going to 
use the system or technology? Will they use it alone or together with others? Besides 
the ‘who’ question, we have to ask why, how, and when the system or technology will be 
used. These questions are central, especially during the evaluation and experimentation 
stage with potential users of the system or technology. UCD methods like brainstorm- 
ing, storyboarding, creating cultural probes, use scenarios and personas, mockups, low- 
and high-fidelity prototypes, and then later when design process progresses to user tests, 
thinking aloud evaluation sessions, focus groups, etc. are all very useful concepts to 
apply when answering these questions. 

In the next section, we will summarize firstly the principles of sociotechnology that 
provide the base for a user-centred design process, e.g., to create embodied interactions to 
increase user experience while interacting with the systems provided. Secondly, we will 
introduce the user-centred design approach by showing their characteristics, especially in 
innovative design processes. In this section, we will explore the role of different kinds of 
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models to facilitate Design Thinking methods in user-centred design processes. Finally, 
we conclude with some discussion points. 


12.2 From sociotechnology as a principle to embodied interaction 


Emery and Trist (1960) introduced the term sociotechnical systems to describe the com- 
plex interaction between humans, machines, and the environment aspects of the work 
system. The goal is to consider people, machines, and context when designing and de- 
veloping such systems. Bedham et al. (2000) described sociotechnical systems as having 
five main characteristics: 


= Systems should have interdependent parts. 
=" Systems should adapt to and pursue goals in external environments. 


=" Systems have an internal environment comprising separate but interdependent tech- 
nical and social subsystems. 


« Systems have equifinality. In other words, systems’ goals can be achieved by more 
than one means. This implies that there are design choices to be made during system 
development. 


=" System performance relies on the joint optimization of the technical and social sub- 
systems. Focusing on one of these systems to the exclusion of the other is likely to 
lead to degraded system performance and utility. 


Baxter and Sommerville (2011) introduced the term of sociotechnical system engi- 
neering to address the need to deliver the expected support for the real work in orga- 
nizations. With sociotechnical system engineering they mean ‘the systematic and con- 
structive use of sociotechnical principles and methods in the procurement, specification, 
design, testing, evaluation, operation and evolution of complex systems’ (p. 4). It is still 
a common problem that systems often meet their technical requirements but are seen by 
their users as failures because they do not deliver the expected support for the real use. 
To avoid producing failure in system engineering, sociotechnical principles and methods 
should be used in design and engineering processes. This can be facilitated by applying 
user experience design methods while designing new systems or interactions, especially 
when it comes to offering engaging and enjoyable interaction for users. ‘Ubiquitous 
computing environments need to be responsive to people's needs, but also need to pro- 
vide engaging and aesthetic experiences' (Banyon, 2019). Besides focusing on usability 
to achieve the best functionality and effective usage of systems, designers must think 
about maximizing users' pleasure while interacting with the systems they design, which 
can also be improved further to facilitate a certain (desirable) lifestyle imposed within 
the design of an object or interaction provided. A successful user experience requires the 
consideration of all senses in the interaction with systems aiming for high usability and 
acceptance by their users. To achieve this, we need embodiment in interaction mecha- 
nisms, at least to a certain degree, if not completely. Embodiment in this sense focuses 
purely on interaction with the objects themselves; as Dourish (2004) explains *... we 
take activity and interaction with the real phenomena of experience to be central, rather 
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than focus on internal or purely cognitive interpretations.’ This shows us that embodied 
interaction does not need any translation (van Rheden & Hengeveld, 2016, p. 349). Its 
specificity embeds meaningful input and output for users. Users control what is relevant 
for the activity. They decide what is needed for the interaction, not the object or the sys- 
tem itself. Of course, one of the goals of designers should be to achieve a high degree of 
seamless embodiment in the interaction, which again requires a precise mapping of bod- 
ily expression to the expression of the device’s output. At best, this results in an arranged 
or coordinated way of acting that is smooth, gentle, and natural. 

The question at the heart of this paper is how to ensure a successfully realized user 
experience in a new design. In other words, how to proceed in a design project in a 
way that understands the target users and their context, including their past experiences, 
and to consider this insight in the design of artifacts and interactions provided as part of 
the new design. My answer to these questions is to apply modeling in all phases of the 
design process by creating models of all findings gathered after studying the target users 
and their past and current contexts, as well as by preparing and accompanying the design 
process as a reflective and self-critical practice. In the next section, we will show which 
models are needed to facilitate a co-creation process in a UCD project, by putting users 
at the centre of attention. 
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Figure 12.1: The iterative process of user-centred design (Adapted from ODIN EN ISO 
9241-210, 2019, p.21) 


12.3 User-Centred Design as a Model-Based Co-Creation Process 


User-centred thinking is about creating a direct link between the current and future users 
(Baek et al., 2007; Wallach & Scholz, 2012). Gould and Lewis (1985) defined three 
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principles for a UCD process: early focus on users and tasks, to gather knowledge about 
the social, cultural, personal, and all other types of characteristics of users; empirical 
measurement, gained by capturing and analyzing user feedback; and iterative design, 
based on iterations after each user feedback. The iterative process of UCD allows for 
approaching a final product step by step, by reducing development risks and avoiding 
dismissing big parts of the achieved components or results (Figure 12.1). 

Besides involving users in design processes, we believe that Design Thinking (Cross 
et al., 1992; Eastman et al., 2001) is a very helpful approach in designing sociotechnical 
systems. “Design thinking is a human-centred approach to innovation that draws from the 
designer’s toolkit to integrate the needs of people, the possibilities of technology, and the 
requirements for business success’ (Tim Brown, IDEO). Design Thinking as a framework 
provides a set of methods that are used in UCD processes. If we take Design Thinking 
as an approach seriously and apply (all) its methods thoroughly throughout the whole 
design process, we can easily follow the goal of understanding everyday practice and its 
actors. This would furthermore lead us to the design of systems that consider the context 
of use, user experiences, and the necessary technology support as a substantial part of the 
sociotechnological approach. Our objective in designing systems is being innovative and 
improving user experience. We think this can only be done by understanding the actors, 
their actions, their use context, and, of course, by including them as experts in the design 
process. 

Exploring the Design Thinking methods that are necessary to set up and carry out a 
UCD process, we end up creating artifacts in each step of the design process (Tellioglu, 
2016, p. 24). These artifacts are both enablers and hosts of the evolving design ideas. 
In the course of design processes, especially if they are user-centric, several models are 
created (Figure 12.2). 
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Figure 12.2: User-centred design in relation to use, system, and interaction models, 
and assigned methods 
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In a UCD process, contextual inquiry, the capturing of ludic experiences, and use 
context definition are needed to describe and understand the use scenarios, and further- 
more to specify the users’ requirements based on their abilities and conditions. Later on, 
designing and evaluating solutions for users requires the design of an interactive user- 
centred system and the following product design. The following models help to support 
these processes: 


= Use models are personas, scenarios, use cases, flow models, storyboards, or nar- 
rative posters, mainly presented as models and descriptions by using a standard 
modelling language like UML (Unified Modelling Language). The aim of these 
models is to detail and describe the design not only for the design team, but also 
to make it accessible for others who are not actively involved in but related to the 
design process. Use models help to address several requirements and answer the 
following design and specification questions: Who are the final users? What are the 
interaction and interface elements? How does the layout, user interface, and inter- 
action look like? What does the user do? What does the product do? What are the 
use scenarios and use contexts? What are the use qualities? What are the specifics 
of the product? What is the positive impact of the product? In which cases does the 
product help users? What are the features of the product? Who would like to have 
the product? Is it feasible? ... 


= System models are interface and interaction visualizations, technology probes as 
well as (hi-fidelity) executable 2D or 3D prototypes showing how the original idea 
looks like in action in the envisioned context. Interaction models are product de- 
scriptions and presentations with final corporate identity elements, demonstrating 
the use and features of the product, pricing, and measures for dissemination. They 
show the idea of the final product or service by referring to its technology features, 
interfaces, architectural elements, and its real time use. System and interaction mod- 
els help designers to deal with the following (re)design, interaction, and evaluation 
questions: What type of layout elements are needed for surfaces, interfaces, colours, 
etc.? What are the dimensions and scales of the product? Are there variations in the 
design? What are the functions that are usable and show affordance? How are er- 
gonomic factors considered in the design? Which technologies should be used to 
implement the idea? Which (embodied) interactions are implemented? Which part 
will be implemented with Wizard of Oz, by just enabling unimplemented technol- 
ogy to be evaluated by using a human to simulate the response of a system? Which 
material, tools, hardware, etc. will be used? What are the sketches, wireframes, 
technology probes, and prototypes? Are there different visualizations? How is the 
product documented? What are the user references and technical documents of the 
product? How are intermediaries or the final product evaluated? What is the evalua- 
tion set-up and what are the points to evaluate? How are evaluation results translated 
into (new) requirements and changes to the existing requirements? ... 


Our claim is that if designers do not create such models, they will fail their design 
purpose. The above-mentioned models offer a holistic view of the design objects and 
processes by helping both the designers and the users. They help address and fulfil the 
requirements for designing (embodied) interactions for better and improved user expe- 
rience. Use models are applied to study and understand the target users, their habits, 
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wishes, mental and physical conditions, settings of use, and desires. Furthermore, they 
enable experimentation with the users to find out how to engage them and how to moti- 
vate them to remain active in the application of the design object. System and interaction 
models continue with designing usable artifacts as parts of the whole system under devel- 
opment, based on the knowledge and experience gathered and provided by use models. 
They facilitate design corrections by means of co-creation processes carried out together 
with target users by ensuring the achievement of maximum usability, clarity, pleasure, 
and satisfaction of users while interacting with the system. Embodiment can be an im- 
portant part of the interaction mechanism designed. Finally, all meaningful input and 
output actions and interactions are realized during the product design by considering 
additional aspects of usage, such as control and customization of the system by its users. 


12.4 Example of a model-based co-creation process 


In this section, we present an example to illustrate how this methodology and the above- 
mentioned models can be applied in a real design project. RRHABITAT-ImmoCHECK+, 
a research project funded by BMK,! has developed the conceptual basis of a gender and 
age sensitive set of instruments for illustrating the development potential of vacant or 
not fully occupied single family detached houses. This potential could be generated by 
redensification and by fostering innovative forms of living together. Furthermore, this 
set of instruments permitted an assessment of the houses. On the one hand, it aimed 
to support persons (the users of the designed product) in a phase of reorientation re- 
garding their living and housing situation; on the other, it provided banks with decision 
guidance in the granting of credits. UCD was applied throughout this project, which 
resulted in a satisfactory and successful solution, both for the residents and for surround- 
ing stakeholders. Figure 12.3 shows the intermediary results (use as well as system and 
interaction models) created in several design workshops with all involved persons in the 
project: a) 2D bricolage of the house; b) a brick presentation of the house; c) emotional 
and behavioural expectations from the house; d) the ground-plan of the house as a Lego 
construction; e) design workshops with users to plan the ‘new’ house; f) the use percep- 
tion of the house and its areas; g) the representation of the ‘new’ house with different use 
aspects; h) the use context mapping based on the usage of the house; i) the visualization 
of the usage of the house from the other inhabitants’ perspective, which varied in most 
cases significantly among the inhabitants of the same house; j) the scenario to implement 
in the technological solution; k) the sketch of an entry point interface; 1) a wireframe and 
low-fi mock-up of the usage representation of the house; m) the prototype of an interac- 
tive solution to create the house plan based on usage; n) the final product landing page 
to enable multiple users to enter data into the system; o) the interface to build a simple 
representation of the house to facilitate a common understanding between all stakehold- 
ers involved. This example shows how the single models build the base for the next step 
in the process, and how the design evolves over time through the active participation of 
the stakeholders. Most importantly, it shows the process by demonstrating the different 
types of models created. 


! Federal Ministry Republic of Austria Climate Action, Environment, Energy, Mobility, Innovation and Tech- 
nology, https://www.bmk.gv.at/en.html 
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Figure 12.3: Artifacts created as models during the User-Centered Design process in 
the project REHABITAT-InmoCHECK^ 
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12.5 Conclusions 


In this paper, we introduced user-centred design as a dynamic multidimensional process 
utilized by several Design Thinking methods and UCD artifacts. The whole design pro- 
cess is an iterative circle of intertwined factors, namely of people (users, designers, other 
stakeholders), particular design phases, and artifacts as intermediaries or final results to 
represent certain design aspects and parameters. The iteration of a UCD process is ac- 
companied with user studies for design and for evaluation, which need different method- 
ological approaches in each design phase. In UCD projects, usability studies have to be 
seen as integral parts of design processes. This fact makes usability studies to important 
activities which enable the creation of the products and shape their future use in real 
settings (Bødker, 2000). 

We presented how a UCD process can be established and how a design process can 
evolve from the very beginning until the definition and presentation of the product design. 
A careful combination of by now well-established Design Thinking methods makes it 
possible to design systems for improved user experience. It remains crucial, however, to 
create the right intermediaries while focusing on users during the design process. The 
generated design artifacts host implicit knowledge about the target users, their contexts, 
and all other factors that are relevant for the design process, mainly to make the right 
design decisions throughout an entire project. Use, system, and interaction models are 
very powerful artifacts that help to achieve this goal when applied correctly at the right 
time while designing. There is no strict rule stating that all the methods contributing to 
create the models presented in Figure 12.2 should be used in any type of design projects. 
Each project is unique, and designers have to select the most suitable methods for their 
particular project. This paper only helps to show the possible and useful ways of doing 
user-centred design. 
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